Using13 Frame13 Seman-cs13 in13 Natural13 Language13 Processing13 13

Apoorv13 Agarwal13 Daniel13 Bauer13 Owen13 Rambow13

Center13 for13 Computa-onal13 Learning13 Systems13 Columbia13 University13

rambowcclscolumbiaedu13

Goal13 of13 Talk13

bull  Claim13 FrameNet13 is13 ready13 to13 be13 used13 in13 NLP13 13 bull  But13 itrsquos13 not13 quite13 like13 using13 a13 POS13 tagger13 bull  Show13 213 case13 studies13

1  Use13 FrameNet13 parser13 (SEMAFOR)13 in13 system13 for13 extrac-on13 of13 social13 network13 from13 narra-ve13 text13

2  Use13 FrameNet13 resource13 (frame13 hierarchy)13 in13 system13 for13 genera-ng13 pictures13 from13 text13 (ldquoseman-c13 groundingrdquo)13

bull  Draw13 some13 conclusions13

Case13 Study13 113 Extrac-on13 of13 Social13 Networks13 from13 Text13

bull  Apoorv13 Agarwal13 bull  Agarwal13 amp13 Rambow13 201013 (EMNLP)13 Agarwal13 et13 al13 201313 (IJCNLP)13 Agarwal13 et13 al13 201413 (EACL)13

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1  Set13 of13 binary13 social13 -es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni-on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1   Set13 of13 binary13 social13 4es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni4on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

Social13 Network13 Extrac-on13 13

bull  Basic13 claim13 social13 network13 made13 up13 of13 social13 events13

bull  Social13 Event13 13 ndash 213 people13 interac4ng13 13 John13 and13 Mary13 had13 dinner13

ndash 113 person13 observing13 another13 13 John13 saw13 Mary13 across13 the13 room13

bull  No13 social13 event13 I13 know13 both13 John13 and13 Mary13 13

Social13 Events13

Interac-on13

Observa-on13

From13 Social13 Events13 to13 Social13 Networks13

bull  Modeling13 social13 events13 ndash 213 people13 =gt13 213 nodes13 ndash unidirec-onal13 or13 bidirec-onal13 link13

bull  Social13 network13 union13 of13 many13 social13 events13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Goal13 of13 Talk13

bull  Claim13 FrameNet13 is13 ready13 to13 be13 used13 in13 NLP13 13 bull  But13 itrsquos13 not13 quite13 like13 using13 a13 POS13 tagger13 bull  Show13 213 case13 studies13

1  Use13 FrameNet13 parser13 (SEMAFOR)13 in13 system13 for13 extrac-on13 of13 social13 network13 from13 narra-ve13 text13

2  Use13 FrameNet13 resource13 (frame13 hierarchy)13 in13 system13 for13 genera-ng13 pictures13 from13 text13 (ldquoseman-c13 groundingrdquo)13

bull  Draw13 some13 conclusions13

Case13 Study13 113 Extrac-on13 of13 Social13 Networks13 from13 Text13

bull  Apoorv13 Agarwal13 bull  Agarwal13 amp13 Rambow13 201013 (EMNLP)13 Agarwal13 et13 al13 201313 (IJCNLP)13 Agarwal13 et13 al13 201413 (EACL)13

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1  Set13 of13 binary13 social13 -es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni-on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1   Set13 of13 binary13 social13 4es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni4on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

Social13 Network13 Extrac-on13 13

bull  Basic13 claim13 social13 network13 made13 up13 of13 social13 events13

bull  Social13 Event13 13 ndash 213 people13 interac4ng13 13 John13 and13 Mary13 had13 dinner13

ndash 113 person13 observing13 another13 13 John13 saw13 Mary13 across13 the13 room13

bull  No13 social13 event13 I13 know13 both13 John13 and13 Mary13 13

Social13 Events13

Interac-on13

Observa-on13

From13 Social13 Events13 to13 Social13 Networks13

bull  Modeling13 social13 events13 ndash 213 people13 =gt13 213 nodes13 ndash unidirec-onal13 or13 bidirec-onal13 link13

bull  Social13 network13 union13 of13 many13 social13 events13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Case13 Study13 113 Extrac-on13 of13 Social13 Networks13 from13 Text13

bull  Apoorv13 Agarwal13 bull  Agarwal13 amp13 Rambow13 201013 (EMNLP)13 Agarwal13 et13 al13 201313 (IJCNLP)13 Agarwal13 et13 al13 201413 (EACL)13

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1  Set13 of13 binary13 social13 -es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni-on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1   Set13 of13 binary13 social13 4es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni4on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

Social13 Network13 Extrac-on13 13

bull  Basic13 claim13 social13 network13 made13 up13 of13 social13 events13

bull  Social13 Event13 13 ndash 213 people13 interac4ng13 13 John13 and13 Mary13 had13 dinner13

ndash 113 person13 observing13 another13 13 John13 saw13 Mary13 across13 the13 room13

bull  No13 social13 event13 I13 know13 both13 John13 and13 Mary13 13

Social13 Events13

Interac-on13

Observa-on13

From13 Social13 Events13 to13 Social13 Networks13

bull  Modeling13 social13 events13 ndash 213 people13 =gt13 213 nodes13 ndash unidirec-onal13 or13 bidirec-onal13 link13

bull  Social13 network13 union13 of13 many13 social13 events13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1  Set13 of13 binary13 social13 -es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni-on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1   Set13 of13 binary13 social13 4es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni4on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

Social13 Network13 Extrac-on13 13

bull  Basic13 claim13 social13 network13 made13 up13 of13 social13 events13

bull  Social13 Event13 13 ndash 213 people13 interac4ng13 13 John13 and13 Mary13 had13 dinner13

ndash 113 person13 observing13 another13 13 John13 saw13 Mary13 across13 the13 room13

bull  No13 social13 event13 I13 know13 both13 John13 and13 Mary13 13

Social13 Events13

Interac-on13

Observa-on13

From13 Social13 Events13 to13 Social13 Networks13

bull  Modeling13 social13 events13 ndash 213 people13 =gt13 213 nodes13 ndash unidirec-onal13 or13 bidirec-onal13 link13

bull  Social13 network13 union13 of13 many13 social13 events13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

ldquoSocial13 Networkrdquo13 Two13 Meanings13

1   Set13 of13 binary13 social13 4es13 between13 people13 ndash  Integral13 part13 of13 our13 cogni4on13 ndash Also13 found13 in13 other13 species13

2  So^ware13 or13 website13 which13 allows13 people13 to13 expand13 and13 manage13 their13 social13 network13 13

Social13 Network13 Extrac-on13 13

bull  Basic13 claim13 social13 network13 made13 up13 of13 social13 events13

bull  Social13 Event13 13 ndash 213 people13 interac4ng13 13 John13 and13 Mary13 had13 dinner13

ndash 113 person13 observing13 another13 13 John13 saw13 Mary13 across13 the13 room13

bull  No13 social13 event13 I13 know13 both13 John13 and13 Mary13 13

Social13 Events13

Interac-on13

Observa-on13

From13 Social13 Events13 to13 Social13 Networks13

bull  Modeling13 social13 events13 ndash 213 people13 =gt13 213 nodes13 ndash unidirec-onal13 or13 bidirec-onal13 link13

bull  Social13 network13 union13 of13 many13 social13 events13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Network13 Extrac-on13 13

bull  Basic13 claim13 social13 network13 made13 up13 of13 social13 events13

bull  Social13 Event13 13 ndash 213 people13 interac4ng13 13 John13 and13 Mary13 had13 dinner13

ndash 113 person13 observing13 another13 13 John13 saw13 Mary13 across13 the13 room13

bull  No13 social13 event13 I13 know13 both13 John13 and13 Mary13 13

Social13 Events13

Interac-on13

Observa-on13

From13 Social13 Events13 to13 Social13 Networks13

bull  Modeling13 social13 events13 ndash 213 people13 =gt13 213 nodes13 ndash unidirec-onal13 or13 bidirec-onal13 link13

bull  Social13 network13 union13 of13 many13 social13 events13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13

Interac-on13

Observa-on13

From13 Social13 Events13 to13 Social13 Networks13

bull  Modeling13 social13 events13 ndash 213 people13 =gt13 213 nodes13 ndash unidirec-onal13 or13 bidirec-onal13 link13

bull  Social13 network13 union13 of13 many13 social13 events13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

From13 Social13 Events13 to13 Social13 Networks13

bull  Modeling13 social13 events13 ndash 213 people13 =gt13 213 nodes13 ndash unidirec-onal13 or13 bidirec-onal13 link13

bull  Social13 network13 union13 of13 many13 social13 events13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Alice13 in13 Wonderland13

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Alice13 in13 Wonderland13

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Other13 Applica-ons13 13

bull  Diploma-c13 cables13 bull  Social13 network13 of13 the13 Taliban13 Government13 of13 Afghanistan13 (as13 described13 13

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13 and13 Seman-cs13

bull  The13 defini-on13 of13 ldquosocial13 eventrdquo13 is13 both13 13 ndash  broad13 (many13 different13 types13 of13 events)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Precise13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Interac4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

Observa4on13

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13 and13 Seman-cs13

bull  Social13 events13 are13 both13 13 ndash  broad13 (many13 different13 types13 of13 event)13 and13 13 ndash  specific13 (requirements13 on13 cogni-ve13 state13 of13 two13 people)13

bull  Specific13 seman-cs13 important13 ndash What13 is13 the13 situa-on13

bull  Buying13 an13 apple13 from13 someone13 13 bull  Seeing13 someone13 across13 a13 room13 13 13 bull  Exceeding13 someone13 in13 height13

ndash What13 are13 the13 roles13 of13 the13 two13 people13 bull  Sanjeev13 talked13 to13 Miriam13 about13 Sam13 and13 about13 Joseacute13

Interac4on13 Observa4on13

-shy‐-shy‐-shy‐13

-shy‐-shy‐-shy‐13

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Social13 Events13 and13 FrameNet13

bull  Hypothesis13 a13 complete13 correct13 FrameNet13 analysis13 would13 be13 helpful13 for13 detec-ng13 and13 classifying13 social13 events13

bull  We13 use13 SEMAFOR13 as13 a13 black13 box13 seman-c13 parser13

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Problem13 113 with13 Using13 FrameNet13 Composi-onality13

bull  FrameNet13 (and13 thus13 SEMAFOR)13 does13 not13 create13 a13 single13 seman-c13 representa-on13 for13 a13 whole13 sentence13

bull  We13 construct13 single13 tree13 from13 spans13 and13 analyses13 contained13 in13 spans13

bull  This13 is13 not13 complex13 unless13 there13 are13 errors13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 drugs13 from13 the13 defendants13 ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

Intoxicant13

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Problem13 213 with13 Using13 FrameNet13 Coverage13

bull  FrameNet13 does13 not13 have13 complete13 lexical13 coverage13 ndash Nor13 does13 SEMAFOR13

bull  Need13 to13 keep13 unanalyzed13 parts13 of13 sentence13 in13 syntac-c13 representa-on13 (=13 deep13 dependency)13

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Single13 Seman-c13 Tree13

Coleman13 said13 he13 bought13 adulterated13 oxy13 from13 the13 defendants13

ROOT

Commerce_buy

Target

4

Buyer

T1-Ind

Seller

from

T2-Grp

Statement

Target

claimed

4

Speaker

T1rsquo-Ind

Message

4

Statement

Speaker

T1-Ind

Coleman

Message

Commerce_buy

Buyer

T1rsquo-Ind

he

Seller

T2-Grp

defendants

Figure 2 Semantic trees for the sentence ldquoColeman claimed [he]T1Ind bought drugs from the[defendants]T2Grprdquo The tree on the left is FrameForest and the tree on the right is FrameTree 4in FrameForest refers to the subtree (bought (T1-Ind) (from T2-Grp)) Ind refers to individual and Grprefers to group

We prune away all subtrees that do not containthe target entities We refer to the resulting treeas FrameForest

For example in Figure 2 the left tree is theFrameForest tree for sentence (5) There are twoframes in this sentence that appear in the final treebecause both these frames contain the target enti-ties and thus are not pruned away The two framesare Commerce_buy and Statement We first cre-ate trees for each of the frames For the Com-merce_buy frame there are three frame elementsTarget (the frame evoking element) Buyer andSeller For each frame element we get the sub-tree from the dependency tree that contains all thewords belonging to that frame element The sub-tree for FEE Target is (bought T1-Ind (from T2-Grp)) The subtree for FE Buyer is (T1-Ind) andthe subtree for FE Seller is (from T2-Grp) Weconnect these subtrees to their respective frame el-ements and connect the resulting subtrees to theframe (Commerce_buy) Similarly we create atree for the frame Statement Finally we connectall frame trees to the ROOT

In this representation we have avoided theframe overlapping issues by repeating the com-mon subtrees the subtree (bought T1-Ind (fromT2-Grp)) is repeated under the FEE Target of theStatement frame as well as under the FE Messageof the Statement frame

452 FrameTree Tree Representation

For the design of this tree we deal with the twooverlapping conditions shown in Figure 1 differ-ently If one frame is fully embedded in another

frame we add the former as a child of the latterframe In Figure 2 the frame Commerce_buy isfully embedded in the frame element Message ofthe frame Statement Therefore the frame sub-tree for Commerce_buy appears as a subtree ofMessage

If the frames overlap partially we copy over theoverlapping portions of the structures to each ofthe frame sub-trees

For the design of this representation we removeall lexical nodes (struck out nodes in Figure 2) andtrees that do not span any of the target entities (notshown in the figure) As a result this structureis the smallest semantic structure that contains thetwo target entities The right tree in Figure 2 is theFrameTree tree for sentence (5)

453 FrameTreeProp Tree RepresentationWe are using a partial tree kernel (PTK) for calcu-lating the similarity of two trees (as detailed in sec-tion 5) The PTK does not skip over nodes of thetree that lie on the same path For establishing anOBS social event between Coleman and the defen-dants all the structure needs to encode is the factthat one target appears as a Speaker and the otherappears in the Message (of the speaker) In Frame-Tree this information is encoded but in an unclearmanner ndash there are two nodes (Commerce_buyand Seller) that come in between the node Mes-sage and T2-Grp

For this reason we copy the nodes labeled withthe target annotations (T1 T2) to all nodes(that are frame elements of a frame) on the pathfrom them to the root in FrameTree We call this

215

Goods13

oxy13

adulterated13

nmod13

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Problem13 313 with13 Using13 FrameNet13 Analysis13 Accuracy13

bull  Seman-c13 parsing13 is13 hard13 bull  SEMAFOR13 makes13 mistakes13 (though13 geing13 bejerhellip)13

bull  Need13 to13 have13 machine13 learning13 approach13 which13 can13 learn13 from13 errorful13 representa-ons13

bull  Solu-on13 use13 tree13 kernels13 (and13 graph13 kernels)13 on13 seman-c13 trees13 (and13 also13 on13 syntac-c13 trees)13

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Some13 Results13

Model13 Detec4on13 Classifica4on13

Precision13 Recall13 F-shy‐Measure13 13

Bag13 of13 Words13 3413 3913 3713 7113

Seman-cs+Rules13 5113 1013 1613 6013

Syntax+ML13 4613 7513 5713 8113

Syntax+Seman-cs+ML13 4813 7413 5913 8313

Data13 ACE13 annotated13 for13 Social13 Events13

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Case13 Study13 213 Text-shy‐to-shy‐Scene13 Genera-on13

bull  Daniel13 Bauer13 Bob13 Coyne13 Julia13 Hirschberg13 Richard13 Sproat13

bull  WordsEye13 system13 Coyne13 amp13 Sproat13 200113 bull  Using13 FrameNet13 for13 WordsEye13 seman-cs13 Rambow13 et13 al13 200213 Coyne13 et13 al13 201013 Coyne13 et13 al13 201113 Bauer13 et13 al13 201213

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Mo-va-on13 Grounded13 Seman-cs13

bull  Grounded13 seman-cs13 applicable13 to13 many13 interes-ng13 NLP13 applica-ons13 13 ndash discourse13 modeling13 virtual13 worlds13 embedded13 systems13 robo-cs13 text-shy‐to-shy‐scene13 genera4onhellip13

bull  Need13 lexical13 seman-c13 theory13 and13 resource13 to13 bridge13 between13 language13 and13 grounded13 seman-cs13 ndash Star-ng13 point13 FrameNet13

bull  Issue13 what13 type13 of13 seman-cs13

2913

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Current13 WordsEye13 System13

Input13 text13 13 the13 very13 large13 silver13 ball13 is13 on13 the13 table13 the13 ground13 is13 shiny13 The13 table13 is13 under13 the13 small13 willow13 tree13 The13 lion13 is13 one13 foot13 in13 front13 of13 the13 table13 the13 lion13 is13 facing13 the13 ball13 it13 is13 cloudy13 13

3013

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Text-shy‐to-shy‐Scene13 Genera-on13 13

bull  Generate13 a13 graphical13 scene13 from13 a13 textual13 descrip-on13 that13 depicts13 the13 content13 of13 the13 descrip-on13

bull  Types13 of13 descrip-ons13 13 ndash  Low-shy‐level13 (primi-ve13 spa-al13 rela-ons)13 13

The13 man13 is13 on13 the13 floor13 He13 is13 kneeling13 13 He13 is13 holding13 the13 sponge13 13 The13 bucket13 is13 near13 the13 man13 13

ndash  High-shy‐level13 13 The13 man13 is13 washing13 the13 floor13

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Levels13 of13 Scene13 Descrip-on13

bull  High-shy‐Level13 13 ndash  Func-onal13 view13 Who13 does13 what13 to13 whom13 ndash  Wash(washerx113 13 themex2)13 13 ndash  Descrip-ons13 involves13 ac-onevent13 verbs13 complex13 en--es13 13

bull  Low-shy‐level13 13 ndash  Realiza-on13 view13 How13 is13 it13 done13 13

(graphical13 what13 does13 it13 look13 like)13 13 ndash  On(figurex113 groundx2)13 Grasp(grasperx113 themex3)13 Reach(reacher13 x113 groundx2)13 Kneel(kneelerx1)13

ndash  Just13 spa-al13 rela-ons13 bull  One13 high-shy‐level13 descrip-on13 rarr13 many13 low-shy‐level13 descrip-ons13 bull  In13 graphics13 genera-on13 low-shy‐level13 descrip-on13 ground13 high-shy‐

level13 descrip-ons13

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Transla-ng13 from13 High-shy‐Level13 Descrip-ons13 to13 Low-shy‐level13 Graphical13 Representa-ons13

bull  Requires13 three13 sources13 of13 knowledge13 13 ndash  Lexical13 Knowledge13 13

bull  Textual13 descrip-on13 to13 high-shy‐level13 seman-c13 representa-on13 13 ndash  Graphical13 Knowledge13 13

bull  Translate13 high-shy‐level13 seman-cs13 into13 low-shy‐level13 graphical13 rela-ons13 13 ndash  Factual13 Knowledge13

bull  Guide13 transla-on13 rule13 out13 impossibleunlikely13 graphical13 representa-ons13 13

bull  Use13 a13 common13 frame-shy‐based13 representa-on13 to13 bridge13 between13 language13 func-onal13 and13 graphical13 meaning13 ndash  ldquoVigNetrdquo13 13

bull  Star-ng13 point13 Frame13 Seman-cs[Fillmore13 1982]13 13

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Lexical13 Knowledge13 FrameNet13 13

bull  Bridge13 language13 and13 high-shy‐level13 seman-c13 representa-on13

bull  Can13 build13 on13 FrameNet13 ndash High-shy‐level13 seman-cs13 13 func-onal13 view13 13 ndash Mapping13 from13 syntax13 13 lexicon13 to13 frame13 seman-cs13 by13 providing13 example13 annota-ons13 for13 each13 frame13 13

ndash  Frame-shy‐to-shy‐Frame13 rela-ons13 13 13 [Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 Subj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Obj13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 PP(for)13 13 13

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Problems13 with13 FrameNet13 Composi-onality13 13

[Mary]buyer13 boughtCommerce13 buy13 [an13 apple]goods13 [for13 $1]money13 13 13 bull  FrameNet13 annota-ons13 are13 lsquoshallowrsquo13 (no13 seman-c13 objects13 as13 arguments13 just13 text13 spans)13 ndash Does13 not13 represent13 seman-cs13 of13 whole13 sentence13 in13 one13 structure13

bull  Does13 not13 represent13 co-shy‐reference13 bull  Solu-on13 no-on13 of13 ldquoinstancerdquo13 of13 a13 frame13 13 ndash Allows13 represen-ng13 seman-cs13 of13 whole13 sentence13 ndash Allows13 co-shy‐reference13

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Instan-a-ng13 Frames13 13 Types13 and13 Instances13 13

bull  Frames13 describe13 concept13 types13 bull  When13 lexical13 items13 evoke13 a13 frame13 in13 a13 descrip-on13 the13 frame13 is13

instan4ated13 bull  All13 frames13 carry13 a13 lsquoselfrsquo13 frame13 element13 which13 is13 bound13 to13 the13 instance13 of13

the13 frame13 13 bull  When13 instan-a-ng13 a13 frame13 bind13 all13 the13 frame13 elements13 to13 instances13

(which13 may13 be13 defined13 by13 another13 frame)13 13

bull  lsquo13 Mary13 bought13 an13 apple13 for13 $1lsquo13 13

13 Commerce13 buy(self13 i613 buyer13 i413 13 13 13 13 13 13 seller13 i113 13 13 13 13 13 13 goods13 i213 13 13 13 13 13 13 13 money13 i3)13 13 13 Person(selfi4)13 Person(selfi1)13 Apple(selfi2)13 Money(selfi313 amounthellip13 )13 13 13 13 13 13

Instantiating Frames Types and Instances

I Frames describe concept types

I When lexical items evoke a frame in a description the frameis instantiated

I All frames carry a lsquoselfrsquo frame element which is bound to theinstance of the frame

I When instantiating a frame bind all the frame elements toinstances (which may be defined by another frame)

Commerce buy(self i6buyer i4seller i1goods i2money i3)

lsquo Mary bought an apple for $1lsquo

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Graphical13 Knowledge13 13

bull  Need13 knowledge13 about13 arrangement13 of13 3D13 models13 to13 depict13 a13 situa-onevent13 13

bull  Low-shy‐level13 seman-cs13 realiza-on13 view13 13 bull  Non-shy‐composi-onality13 of13 verb13 meaning13 13 ndash Correct13 visualiza-on13 of13 verb13 depends13 on13 verb13 and13 its13 arguments13 13

Graphical Knowledge

I Need knowledge about arrangement of 3D models to depict asituationevent

I Low-level semantics realization viewI Non-compositionality of verb meaning

I Correct visualization of verb depends on verb and itsarguments

lsquoThe man washed the floorlsquo lsquoThe man washed the applelsquo

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Many13 Wash13 Op-ons13

3813

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Note13 Lexicaliza-on13 Differences13 13 Across13 Languages13

bull  Verbs13 for13 lsquowashrsquo13 donrsquot13 simply13 mean13 lsquoremove13 dirtrsquo13 EnglishFrenchEgyp-an13 Arabic13 ndash  John13 washed13 the13 applelavergasal13 ndash  John13 washed13 the13 floorlaver13 neIoyermasaH13 ndash  John13 brushed13 his13 teethlavergasal13

bull  brush13 encodes13 the13 instrument13

bull  Support13 verb-shy‐noun13 construc-ons13 o^en13 not13 composi-onal13 (Persian)13

13

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Graphical13 Knowledge13 Vignejes13 13 Graphical Knowledge Vignettes

[Coyne et al 2011]

I Frames with decomposition grounded in graphical primitives

I Represent dicrarrerent realizations for lexical framesI Vignettes extend frames by

I optionally introducing new frame elements that participate inthe visualization

I decomposition into sub-framesI link to specific 3D model types (frames describing entities)I describe graphical structure of a scene (frames describing

eventssituations)

Commerce counter( buyer goods money seller)

(ISA) Commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

bull  Vignejes13 extend13 frames13 by13 1  13 Adding13 new13 specialized13 frames13 (extend13 frame13 hierarchy)13 based13 on13

selec-on13 restric-ons13 for13 frame13 elements13 2  Op-onally13 introducing13 new13 frame13 elements13 that13 par-cipate13 in13 the13

visualiza-on13 3  Decomposing13 into13 sub-shy‐frames13 13

bull  link13 to13 specific13 3D13 model13 types13 (frames13 describing13 en--es)13 bull  describe13 graphical13 structure13 of13 a13 scene13 (frames13 describing13 eventssitua-ons)13 13 13

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Graphical13 Knowledge13 13 Vigneje13 Decomposi-on13 13

Graphical Knowledge Vignette Decomposition

I (temporal) subframe relation in FrameNet

I New frame-to-frame relation subframe parallel

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Factual13 (World)13 Knowledge13 13

bull  Some13 ontological13 informa-on13 already13 encoded13 in13 frame-shy‐to-shy‐frame13 rela-ons13 (inheritance)13 13

bull  In13 addi-on13 frame13 defini-ons13 for13 en-ty13 types13 need13 13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 non-shy‐graphical13 proper-es13 of13 objects13 13 ajributes13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 informa-on13 about13 parts13 ndash  1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 13 world13 knowledge13 (lsquoapples13 grow13 on13 treeslsquo13 `apples13 are13 bought13 in13 stores13 or13 marketsrsquo)13

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounded13 seman-cs13 through13 vignejes13 4313

The13 man13 washes13 the13 stage13 with13 a13 sponge13

Subj13 Obj13

Adj13

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounded13 seman-cs13 through13 vignejes13 4413

The13 man13 washes13 the13 stage13 with13 a13 sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounded13 seman-cs13 through13 vignejes13 4513

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounded13 seman-cs13 through13 vignejes13 4613

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounded13 seman-cs13 through13 vignejes13 4713

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounded13 seman-cs13 through13 vignejes13 4813

The13 man13 washes13 the13 stage13 with13 a13 sponge13

man13

stage13

sponge13 washe13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

evokes13

SpongeTool(self)13

hellip13

Stage(self)13

Floor(selfself)13 hellip13

Man(self)13

Human(selfself)13 Male(selfself)13 hellip13

Subj13 Obj13

Adj13

instance13

instance13 instance13

instance13

evokes13

evokes13

evokes13

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounded13 seman-cs13 through13 vignejes13 4913

man13 stage13

Washe13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

Wash_floor_w_sponge(self13 washer13 instrument13 theme)13 13

Human(selfwasher)13 Floor(selftheme)13 SpongeTool(selfinstrument)13

On(figurewasher13 ground13 ground)13 Grasp(grasperwasher13 themeinstrument)13 Reach(reacherwasher13 themetheme)13 Kneel(kneelerwasher)13 13

Decomposi-on13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounded13 seman-cs13 through13 vignejes13 5013

man13 stage13

The13 man13 washes13 the13 stage13 with13 a13 sponge13

On(self13 figure13 ground)13

hellip13

Grasp(self13 grasper13 theme)13 Reach(self13 reacher13 theme)13

hellip13

Kneel(self13 kneeler)13

hellip13

onr13 kneelr13 graspr13 reachr13

sponge13

hellip13

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Status13 of13 the13 VigNet13 Resource13 13

VigNet13 currently13 contains13 13 bull  a13 small13 set13 of13 primi-ve13 spa-al13 rela-ons13 (on13 next-shy‐to13

(direc-on13 and13 distance)13 in13 direc-on)13 13 bull  small13 set13 (about13 30)13 lsquoabstractlsquo13 vignejes13 13

ndash  holdingtouching13 target13 or13 pa-ent13 using13 handheld13 13 bull  instruments13 using13 sta-onary13 machine13 human13 poses13 13 bull  several13 hundred13 verbal13 vignejes13 inheri-ng13 from13 and13

parameterizing13 abstract13 vignejes13 (ongoing)13 bull  about13 200013 nominal13 vignejes13 mapping13 to13 about13 300013 3D13

models13 (with13 physical13 ajributes13 parts13 affordances)13 13 bull  about13 8013 loca-on13 vignejes13 (all13 rooms13 including13 fixtures

affordances)13 13

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Summary13

bull  FrameNet13 used13 successfully13 in13 two13 applica-ons13 13 ndash Social13 network13 extrac-on13 ndash Text-shy‐to-shy‐scene13 genera-on13

bull  Good13 level13 of13 abstrac-on13 bull  Buthellip13

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Conclusion13 from13 Both13 Case13 Studies13

bull  FrameNet13 comes13 out13 of13 a13 lexicographic13 tradi-on13 bull  NLP13 is13 needs13 seman-c13 representa-ons13 based13 on13 FrameNet13 not13 just13 lexical13 entries13 with13 examples13

bull  Great13 interest13 in13 sophis-cated13 composi-onal13 (lexical)13 seman-c13 representa-ons13 now13 (AMR)13

bull  Need13 for13 FrameNet13 to13 define13 such13 a13 representa-on13 (this13 is13 not13 super13 complicated)13

bull  NLP13 needs13 annotated13 FrameNet-shy‐based13 seman-c13 representa-on13

bull  FrameNet13 parsers13 should13 produce13 trees13 not13 annotate13 spans13

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Thank13 You13

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Factual13 Knowledge13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Frame13 decomposi-ons13 are13 declara-ve13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Simultaneously13 define13 proper-es13 of13 frame13 element13 fillers13 and13 13 restrict13 fillers13 to13 instances13 of13 frames13 that13 define13 this13 property13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Can13 create13 frame13 elements13 for13 proper-es13 1048576104857710485781048579104858010485811048582104858310485841048585104858610485871048588104858910485901048591104859210485931048594104859510485961048597104859810485991048600104860110486021048603104860410486051048606104860710486081048609104861010486111048612104861310486141048615104861610486171048618104861910486201048621104862210486231048624104862510486261048627104862810486291048630104863110486321048633104863410486351048636104863710486381048639104864010486411048642104864310486441048645104864610486471048648104864910486501048651104865210486531048654104865510486561048657104865810486591048660104866110486621048663104866410486651048666104866710486681048669104867010486711048672104867310486741048675104867610486771048678104867910486801048681104868210486831048684104868510486861048687104868810486891048690104869110486921048693104869410486951048696104869710486981048699104870010487011048702104870313 Or13 use13 lsquoselfrsquo13 frame13 element13 to13 define13 proper-es13 of13 frames13 for13 13 en-ty13 types13 13 13

Factual KnowledgeI Frame decompositions are declarative

I Simultaneously define properties of frame element fillers andrestrict fillers to instances of frames that define this property

I Can create frame elements for propertiesI Or use lsquoselfrsquo frame element to define properties of frames for

entity types

commerce counter( buyer goods money seller)

size(figuregoods sizesmall)animate(selfseller)animate(selfbuyer)(ISA) commerce buyat counter(partcpt1buyer partcpt2seller counterc)on(figuregoods groundc)on(figuremoney groundc)

apple( )

(ISA) fruitsize(figureself sizesmall)shape(figureself shaperound)

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Examples13 of13 highlow13 level13 descrip-ons13 (via13 Mechanical13 Turk)13

5613

Low-shy‐level13 13 A13 man13 is13 using13 the13 telephone13 The13 man13 is13 wearing13 a13 yellow13 vest13 The13 man13 has13 blonde13 hair13 The13 man13 has13 white13 skin13 A13 white13 rodent13 is13 inside13 a13 cage13 The13 cage13 is13 on13 a13 table13 The13 phone13 is13 on13 the13 table13 The13 cage13 has13 a13 handle13 A13 safe13 is13 in13 the13 background13 of13 the13 room13 13 High-shy‐level13 The13 man13 is13 a13 scien-st13 working13 with13 white13 rodents13 13 High-shy‐level13 The13 man13 is13 talking13 to13 another13 scien-st13 13 High-shy‐level13 The13 man13 feels13 guilt13 at13 imprisoning13 a13 white13 rodent13 13

bull  Acquire13 typical13 language13 (hilow)13 for13 10013 comic13 book13 scenes13

bull  Each13 scene13 described13 by13 513 different13 Turkers13

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Goal13

bull  Automa-c13 conversion13 from13 text13 to13 3D13 scene13 ndash Text13 egrave13 seman-cs13 (FrameNet13 amp13 seman-c13 parsing)13

ndash High-shy‐level13 seman-cs13 egrave13 low-shy‐level13 seman-cs13 ndash Low-shy‐level13 seman-cs13 egrave13 3D13 scene13 (WordsEye)13

5713

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Decomposing13 Meaning13 of13 Wash13 13

Agent13 HOLDS13 sponge13 Agent13 NEAR13 bucket13 Agent13 KNEEL13 on13 floor13

Agent13 FACING13 sink13 Agent13 HOLD13 pa-ent13 (not13 shown)13 Agent13 IN-shy‐FRONT-shy‐OF13 sink13

5813

The13 man13 washed13 the13 floor13 13 13 13 13 13 13 13 13 13 The13 man13 washed13 the13 apple13 13

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Grounding13 of13 ldquoofrdquo13

CONTAINMENT13 bowl13 of13 cats13 PART-shy‐OF13 head13 of13 the13 cow13 SIZE-shy‐OF13 height13 of13 horse13 is13

ARRANGEMENT-shy‐OF13 stack13 of13 cats13 SUBSTANCE-shy‐OF13 horse13 of13 stone13 REPRESENTING13 Picture13 of13 the13 girl13 5913

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6013

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Represen-ng13 and13 Grounding13 Meaning13

bull  Seman4c13 nodes13 denote13 en--es13 and13 types13 bull  Types13 (House13 teenager13 polar13 bear13 hellip)13 bull  Elaborated13 types13 (A13 Gny13 house13 with13 a13 metallic13 front13 door)13 bull  Events13 (John13 washing13 the13 apple)13 bull  Individuals13 (Barack13 Obama13 Homer13 Simpson13 my13 house)13

bull  Seman4c13 rela4ons13 are13 applied13 to13 seman-c13 nodes13 to13 give13 them13 meaning13 13

bull  HABITAT-shy‐OF(polar-shy‐bear13 Arc-c)13 bull  SIZE-shy‐OF(house-shy‐113 -ny)13 bull  WASH(JOHN13 APPLE)13 bull  Need13 inventory13 of13 seman-c13 rela-ons13 (FrameNet13 frames)13

bull  Meta-shy‐rela4ons13 translate13 between13 seman-c13 rela-ons13 13 bull  Used13 to13 decomposeground13 meaning13 with13 Vignejes13 (FrameNet13 f2f13 rela-ons)13

6113

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Meta-shy‐rela-ons13 to13 invoke13 and13 ground13 Vignejes13

bull  Vignees13 are13 seman-c13 rela-ons13 represen-ng13 different13 ways13 of13 grounding13 seman-cs13 13

bull  Vignejes13 inherit13 (via13 meta-shy‐rela-ons)13 from13 parent13 rela-on13 ndash  WASH13 egrave13 WASH-shy‐SMALL-shy‐FRUIT13 ndash  Add13 new13 seman-c13 roles13 as13 needed13 for13 grounding13 (eg13 sink)13 ndash  Selec-onal13 restric-ons13 constrain13 types13 of13 the13 seman-c13 roles13

bull  WASH-shy‐SMALL-shy‐FRUIT13 SizeOf(PATIENT13 small)13

ndash  Retain13 high-shy‐level13 meaning13 and13 seman-c13 roles13 from13 parent13 bull  Vignejes13 decompose13 into13 grounded13 rela-ons13 using13 meta-shy‐rela-ons13

ndash  WASH-shy‐SMALL-shy‐FRUIT(man13 apple)13 egrave13 bull  IN-shy‐FRONT-shy‐OF(man13 sink)13 bull  HOLDS(man13 apple)13 bull  FACING(man13 sink)13 bull  REACH-shy‐TOWARD(man13 sink)13

6213

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Represen-ng13 and13 grounding13 meaning13

6313

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Represen-ng13 and13 grounding13 meaning13

6413

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Represen-ng13 and13 grounding13 meaning13

6513

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Represen-ng13 and13 grounding13 meaning13

6613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13

Decompose13 vigneje13 into13 grounded13 rela-ons13 Apply13 constraints13 and13 render13 13

Create13 seman-c13 nodes13 and13 rela-ons13

Invoke13 vigneje13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

6813

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Frame13 Seman-cs13 [Fillmore13 1982]13 13 FrameNet13 [Baker13 et13 al13 199813 Ruppenhofer13 et13 al13 2010]13 bull  Word13 meaning13 can13 only13 be13 understood13 by13 referring13 to13 conceptual13 structure13 evoked13 by13 it13 13 13

bull  Frames13 are13 cogni-ve13 schemas13 describing13 rela-ons13 between13 state13 or13 event13 par-cipants13 13

bull  Role13 of13 syntax13 and13 lexicon13 13 bull  Valence13 pajerns13 of13 a13 lexical13 item13 map13 syntac-c13 arguments13 to13 frame13 elements13 13

bull  FrameNet13 Lexicographic13 resource13 containing13 frames13 their13 rela-ons13 and13 linguis-c13 realiza-ons13

bull  Can13 we13 use13 FrameNet13 to13 ground13 word13 meaning13 via13 conceptual13 structureframes13 13

FrameNet13 and13 its13 limita-ons13 6913

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Problem13 113 FrameNet13 frames13 are13 too13 general13 bull  walk13 and13 swim13 are13 grouped13 in13 the13 Self_Mo-on13 frame13 but13

have13 different13 graphical13 realiza-ons13 13 bull  Wash13 an13 apple13 hellip13 the13 floor13 hellip13 the13 car13 hellip13 hands13 hellip13 hair13 etc13

have13 different13 graphical13 realiza-ons13 13

FrameNet13 and13 its13 limita-ons13 7013

Problem13 213 FrameNet13 frames13 have13 lijle13 internal13 structure13

bull  Frames13 are13 only13 defined13 by13 a13 list13 of13 Frame13 Elements13 (possibly13 with13 seman-c13 types)13 frame-shy‐to-shy‐frame13 rela-ons13 and13 an13 informal13 plaintext13 defini-on13

bull  No13 grounding13 Need13 framework13 to13 decompose13 frames13 into13 (graphical)13 primi-ves13

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Problem13 313 FrameNet13 analyses13 of13 sentences13 are13 ldquoshallowrdquo13

bull  In13 annota-ons13 frame13 elements13 are13 bound13 to13 text13 spans13 not13 seman-c13 objects13

FrameNet13 and13 its13 limita-ons13 7113

Problem13 413 FrameNet13 contains13 only13 lexical13 knowledge13

bull  Only13 source13 of13 conceptual13 knowledge13 (very13 limited13 and13 rigid)13 seman-c13 types13 frame-shy‐to-shy‐frame13 rela-ons13

bull  No13 world13 knowledge13 whatsoever13 ndash  But13 can13 use13 frames13 to13 assert13 world13 knowledge13 (once13 problem13 313 is13 solved)13

typical_habitat(lifeForm13 Sloth13 habitat13 Rain_Forrest)13 13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 types13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Using13 VigNet13 and13 conclusion13

7213 Grounded13 seman-cs13 through13 vignejes13

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Frame13 Decomposi-ons13 and13 Selec-onal13 Restric-ons13 13

bull  Need13 to13 decompose13 frames13 into13 subframes13 13 ndash  Spa-al13 decomposi-on13 of13 loca-onsac-onscomplex13 objects13

ndash  Express13 conceptual13 knowledge13 bull  ldquoApples13 are13 small13 round13 edible13 and13 sweet13 fruitsrdquo13 bull  Use13 for13 complex13 selec-onal13 restric-ons13 on13 frame13 elements13

ldquosomething13 ediblerdquo13 ldquosomething13 small13 and13 roundrdquo13 ldquoa13 sweet13 fruitrdquo13 13

bull  Decomposi-on13 mechanism13 13 ndash  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13 ndash  New13 Self13 frame13 element13 for13 each13 frame13 (assert13 conceptual13 knowledge)13

Grounded13 seman-cs13 through13 vignejes13 7313

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Frame13 Decomposi-ons13

bull  FrameNetrsquos13 subframe13 and13 precedes13 rela-on13

13 bull  New13 frame-shy‐to-shy‐frame13 rela-on13 Subframe_parallel13

13 13 13 13

Grounded13 seman-cs13 through13 vignejes13 7413

Wash-shy‐Apple13

Front-shy‐Of13

Facing13 Grasp13

Reach13

Wash_apple13 (self13 washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

Criminal_Process13

Arrest13 Trial13 13

Sentencing13 13

subframe13

precedes13

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Seman-c13 Nodes13

bull  Problems13 ndash  Frames13 are13 conceptual13 descrip-ons13 frame13 elements13 are13 just13 `variablesrsquo13

ndash  In13 annota-ons13 frame13 elements13 are13 filled13 with13 text13 spans13

ndash  But13 scenes13 are13 composed13 of13 concrete13 objects13 and13 rela-ons13 13

bull  Frames13 are13 instan-ated13 to13 seman-c13 nodes13 bull  Seman-c13 nodes13 are13 ajached13 to13 frame13 elements13 bull  lsquoselfrsquo13 frame13 element13 points13 to13 instance13 of13 its13 frame13

Grounded13 seman-cs13 through13 vignejes13 7513

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Seman-c13 Nodes13 and13 Frames13 13

bull  Seman-c13 Nodes13 13 ndash Frame13 instances13 types13 of13 objects13 or13 rela-ons13 ndash Two13 types13 13

1  Discourse13 referents13 (existen-ally13 quan-fied)13 2  Seman-c13 classes13 for13 knowledge13

representa-on(universally13 quan-fied)13

bull  Frames13 13 ndash  `Constructorsrsquo13 for13 seman-c13 nodesintensionsdescrip-ons13 of13 concepts13

Grounded13 seman-cs13 through13 vignejes13 7613

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 Vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

7713

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Popula-ng13 VigNet 13 13

bull  Use13 and13 extend13 FrameNet13 itself13 13 ndash  Add13 frames13 for13 all13 lexical13 items13 ndash  Add13 new13 graphically-shy‐oriented13 frames13 for13 grounding13 ndash  Add13 valence13 pajerns13 from13 VerbNet13 etc13 ndash  Improve13 linguis-c13 coverage13 using13 corpora13

bull  Import13 WordsEye13 dic-onary13 and13 object13 info13 bull  Acquire13 knowledge13 with13 Mechanical13 Turk13 13 bull  Mine13 rela-onal13 knowledge13 from13 corpora13 and13 exis-ng13 resources13

bull  Add13 relevant13 knowledge13 by13 hand13 ndash  Define13 abstract13 vignejes13 with13 parameterized13 decomposi-ons13

7813

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

BampW13 drawings13 13 13 13 13 13 13 Texture13 Maps13

Artwork13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 Photographs13

3D13 objects13

Import13 WordsEye13 Dic-onary13

2D13 Images13 and13 textures13

7913

bull  220013 3D13 objects13 and13 1000013 images13 represented13 by13 seman-c13 nodes13 ndash  Many13 specialized13 subtypes13 and13 compound13 objects13 (eg13 a13 goat13 head13 mounted13 on13 a13 plank13 of13 wood)13 ndash  Seman-c13 rela-ons13 between13 these13 nodes13 (CONTAINING13 IS-shy‐A13 COLOR-shy‐OF13 HANDLE-shy‐OF13 PART-shy‐OF13 hellip)13

bull  1500013 lexical13 items13 corresponding13 to13 these13 nodes13 (including13 related13 terms13 and13 ajributes)13

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Acquire13 Knowledge13 with13 Mechanical13 Turk13 (Masoud13 Rouhizadeh13 Margit13 Bowler13 Jack13 Crawford)13 (cite13 Masoudrsquos13 paper)13

bull  Default13 loca-ons13 and13 parts13 ndash  Show13 picture13 of13 3D13 objects13 Turkers13 provide13 loca-onsparts13

13

13 bull  Loca-on13 Vignejes13 13 13 13 13 13 13

13 13 13 13 13 ndash  Show13 picture13 of13 different13 types13 of13 rooms13 Turker13 does13

1  Iden-fies13 main13 objects13 2  Specifies13 spa-al13 rela-ons13 for13 those13 objects13 3  Iden-fies13 subobject13 types13 using13 3D13 library13 (eg13 kitchen13 table13 versus13 picnic13 table)13

bull  Ac-on13 Vignejes 13 13 13 13 ndash  Provide13 ac-on13 sentence13 and13 loca-on13 vigneje13 pictureobjects13 Turker13 does13

1  Iden-fies13 addi-onal13 objects13 and13 par-cipants13 2  Iden-fies13 subobject13 types13 using13 3D13 library13 3  Specifies13 spa-al13 rela-ons13 facial13 expressions13 and13 poses13 for13 those13 objects13

8013

CONTAINING(container=SCHOOLHOUSE13 contents=LOCKER)13 13 CONTAINING(container=SCHOOLHOUSE13 contents=DESK)13 CONTAINING(container=SCHOOLHOUSE13 contents=BLACKBOARD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BUSH)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=BIRD)13 HABITAT-shy‐OF(habitat=MEADOW13 inhabitant=WILDFLOWER)13

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Inferring13 Rela-ons13 from13 Corpora13 Sproat13 (2001)13 Inferring13 the13 environment13 in13 a13 text-shy‐to-shy‐scene13 conversion13 system13

8113

Tuple13 Target13 term13 Read13 newspapers13 morning13 Eat13 breakfast13 Morning13 Drink13 tea13 Evening13 Sleep13 on13 floor13 Night13 Look13 in13 mirror13 Morning13 Celebrate13 Easter13 Morning13 Play13 trumpet13 Night13 Eat13 lunch13 A^ernoon13 Go13 to13 bed13 Night13 Eat13 dinner13 evening13 Cook13 meal13 evening13 Take13 shower13 Morning13 See13 boat13 a^ernoon13 Rake13 leaves13 a^ernoon13 Sleep13 in13 chair13 Noon13 Throw13 egg13 Morning13 Take13 to13 hills13 Morning13 Sleep13 in13 bed13 night13

bull  Acquire13 default13 loca-ons13 -mes13 and13 seasons13 for13 ac-ons13 ndash Target13 terms13 rooms13 (eg13 bedroom)13 -me13 of13 day13 (eg13 morning)13 season13 (eg13 winter)13 ndash Corpora13 of13 41513 million13 words13 of13 English13 text13 with13 POS13 tagging13 ndash Extract13 tuples13 for13 verb-shy‐object13 (eg13 wash13 face)13 and13 verb-shy‐prep-shy‐object13 (get13 into13 bed)13 u-lizing13 POS13 tags13 ndash Compute13 associa-on13 between13 the13 tuples13 and13 each13 of13 the13 target13 terms13

Tuple13 Target13 term13 Wash13 clothes13 Laundry13 room13 Wash13 hands13 Bathroom13 Drive13 car13 Garage13 Go13 to13 bathroom13 Bathroom13 Brush13 teeth13 Bathroom13 Run13 car13 Garage13 Wash13 dishes13 Kitchen13 Go13 to13 store13 Laundry13 room13 Go13 to13 bed13 Bedroom13 Take13 shower13 Bathroom13 See13 in13 kitchen13 Kitchen13 Sit13 on13 sofa13 Bedroom13 Sit13 on13 bed13 Bedroom13 Sit13 on13 toilet13 Bathroom13 Sit13 at13 table13 Kitchen13 Hold13 knife13 Bedroom13 Climb13 over13 wall13 bedroom13 Sit13 on13 floor13 hallway13

Loca-ons13 Time13 of13 day13

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Outline13

bull  Mo-va-on13 and13 Goal13 bull  Basic13 Approach13 ndash  Seman-c13 nodes13 seman-c13 rela-ons13 and13 vignejes13

bull  Formalizing13 Vignejes13 for13 grounded13 seman-cs13 bull  FrameNet13 and13 its13 limita-ons13 13 bull  Adding13 Vigneje13 Seman-cs13 to13 FrameNet13

bull  Popula-ng13 VigNet13 bull  Conclusion13

8213

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Note13 Verb13 Homophony13 and13 Polysemy13

bull  FE13 disambiguates13 homophonous13 verb13 senses13 ndash The13 pilot13 banked13 to13 the13 right13 before13 the13 crash13 ndash The13 pilot13 banked13 at13 Chase13 before13 the13 crash13

bull  FE13 disambiguates13 polysemous13 verb13 senses13 ndash Patricia13 glared13 over13 at13 me13 ndash The13 sun13 glared13 down13 on13 us13

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Conclusion13 ndash13 whatrsquos13 new13 bull  Vigneje13 Seman-cs13 ndash13 a13 grounded13 lexical13 seman-c13 theory13 based13 on13 FrameNet13 ndash  Seman-c13 nodes13 represen-ng13 types13 and13 individuals13 ndash  FrameNet13 frames13 provide13 inventory13 of13 seman-c13 rela-ons13 that13 can13 be13

applied13 to13 seman-c13 nodes13 to13 express13 meaning13 ndash  Vignejes13 are13 seman-c13 rela-ons13 that13 bridge13 between13 high-shy‐level13 and13

low-shy‐level13 seman-c13 rela-ons13 (via13 F2F13 meta-shy‐rela-ons)13 13 bull  Up13 Selec-onal13 restric-ons13 leveraging13 seman-c13 nodes13 and13 rela-ons13 bull  Down13 Subframe_parallel13 frame13 decomposi-on13

bull  A13 lexical-shy‐seman-c13 resource13 VigNet13 under13 development13 ndash  Various13 techniques13 for13 popula-ng13 Vignet13

bull  Applying13 VigNet13 in13 text-shy‐to-shy‐scene13 conversion13 8413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

13 8513

coynecscolumbiaedu13 Bauercscolumbiaedu13

rambowcclscolumbiaedu13

This13 material13 is13 based13 upon13 work13 supported13 by13 the13 Na-onal13 Science13 Founda-on13 under13 Grant13 NoIIS-shy‐090436113 13

Thank13 You13

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Represen-ng13 and13 grounding13 meaning13

8613

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Represen-ng13 and13 grounding13 meaning13

8713

Input13 The13 man13 washed13 the13 blue13 Gle13 floor13 Create13 seman-c13 nodes13 and13 rela-ons13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 13

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Leverage13 knowledge13 and13 abstract13 vignejes13 to13 increase13 coverage13

bull  John13 picked13 the13 apple13 ndash  Use13 ldquopick-shy‐from-shy‐treerdquo13 vigneje13 based13 on13 selec-onal13 restric-on13 of13 apple13 being13 from13 a13 tree13

ndash  Use13 knowledge13 to13 determine13 a)13 type13 of13 tree13 and13 b)13 loca-on13 ndash  OriginOf(en-tyapple13 sourceapple-shy‐tree)13 ndash  Habitat(en-tyapple-shy‐tree13 loca-onfield)13

ndash  Decompose13 to13 ndash  InFrontOf(figureJohn13 ground13 apple-shy‐tree)13 ndash  ReachUp(agentJohn13 target13 apple)13 ndash  Ajached(figureapple13 groundapple-shy‐tree13 part13 branch)13 ndash  LocatedIn(figureapple-shy‐tree13 groundfield)13

8813

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

WordsEye13 Create13 Scenes13 from13 Low-shy‐Level13 Descrip-ons13

Coyne13 Sproat13 ldquoWordsEye13 An13 Automa-c13 Text-shy‐to-shy‐Scene13 Conversion13 Systemrdquo13 SIGGRAPH13 200113 Online13 system13 at13 hjpbitlywordseye13

Input13 text13 Santa13 Claus13 is13 on13 the13 white13 mountain13 range13 He13 is13 blue13 It13 is13 cloudy13 A13 large13 yellow13 illuminator13 is13 in13 front13 of13 him13 The13 alien13 is13 in13 front13 of13 him13 The13 mountain13 range13 is13 shiny13

8913

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounding13 meaning13 through13 Vignejes13

9013

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13

Create13 seman-c13 nodes13 and13 rela-ons13

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Grounding13 meaning13 through13 Vignejes13 (spend13 -me)13

9113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Vigneje13 decomposi-on13 Apply13 constraints13 and13 render13 (contextual13 objects13 added)13

Create13 seman-c13 nodes13 and13 rela-ons13

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Represen-ng13 Word13 Meaning13

9213

polar13 bear13 a13 white13 colored13 bear13 that13 lives13 in13 the13 Arc-c13 13

Seman-c13 node13 for13 ldquopolar13 bearrdquo13

Seman-c13 rela-on13 giving13 meaning13 to13 seman-c13 node13 for13 polar13 bear13

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

hellipand13 sentence13 meaning13

9313

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Seman-c13 nodes13 for13 a13 par-cular13 polar13 bear13 and13 fish13

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Represen-ng13 sentence13 meaning13

9413

The13 old13 gray13 polar13 bear13 ate13 the13 fish13

Use13 meta-shy‐rela-ons13 to13 do13 grounding13

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Represen-ng13 Word13 Meaning13

bull  teenager13 human13 between13 1313 and13 1913 years13 old13 13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 9513

bull  polar13 bear13 a13 bear13 with13 white13 fur13 that13 lives13 in13 arc-c13 13

Seman4c13 Nodes13

Seman4c13 Nodes13

Seman4c13 Rela4ons13

Seman4c13 Rela4ons13

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Using13 Seman-c13 Nodes13

bull  Seman-c13 nodes13 can13 be13 ndash  Temporary13 or13 permanently13 stored13 in13 a13 knowledge13 base13 ndash  Linked13 to13 lexicon13 discourse13 or13 `anonymousrsquo13

bull  Asser-ng13 world13 knowledge13 example13 ndash  sn_1512313 Distance(self13 sn_1512313 groundearth13 figuremoon13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 distance13 sn_1314)13 ndash  sn_131413 13 13 Range(self13 sn_131413 min13 400000km13 max13 500000km)13 13

Grounded13 seman-cs13 through13 vignejes13 9613

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Conceptual13 Issues13

bull  What13 goes13 in13 frames13 Conceptual13 vs13 world13 knowledge13 issues13

bull  Deducing13 vigneje13 structure13 from13 noisy13 field13 data13 Or13 designing13 structure13 similar13 to13 FrameNet13

bull  Composing13 ac-on13 vignejes13 with13 loca-on13 vignejes13 13

9713 Using13 VigNet13 and13 conclusion13

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Whatrsquos13 new13 about13 VigNet13 Related13 work13

bull  Knowledge13 representa-on13 Ontologies13 ndash Resources13 ConceptNetOpenMind13 LabelMe13 Cyc13 ndash Theory13 FOL13 Descrip-on13 Logics13

bull  Grounded13 seman-cs13 ndash Event13 Logics13 Conceptual13 Dependency13 PARac-onary13

bull  Lexical13 seman-c13 resources13 ndash FrameNet13 Verbnet13 WordNet13 Ontonotes13 13

bull  Text-shy‐to-shy‐scene13 9813

hjpwwwcscolumbiaeduspeechtext2scenebibliographyhtml13

Using13 VigNet13 and13 conclusion13

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Using13 VigNet13 Vignejes13 corresponding13 to13 different13 senses13 of13 ldquoOfrdquo13 using13 selec-onal13 restric-ons13

9913

Seman-c13 types13 func-onal13 proper-es13 and13 spa-al13 tags13 used13 to13 resolve13 seman-c13 rela-on13 for13 ldquoofrdquo13

Text13 (A13 of13 B)13 Condi4ons13 Seman4c13 Rela4on13 VigneIe13 Decomposi4on13

Bowl13 of13 cherries13

A=container13 B=plurality-shy‐or-shy‐mass13

CONTAINER-shy‐OF13 (bowl13 cherries)13 A13 contains13 B13

Slab13 of13 concrete13

A=en-ty13 B=substance13

MADE-shy‐OF13 (slab13 concrete)13 A13 HAS-shy‐TEXTURE13 B13

picture13 of13 girl13 A=represen-ng13 B=en-ty13

REPRESENTS13 (picture13 girl)13 A13 HAS-shy‐TEXTURE13 C13 C13 REPRESENTS13 B13

Arm13 of13 the13 chair13

A=part-shy‐of(B)13 B=en-ty13

PART-shy‐OF13 (chair13 arm)13

Height13 of13 the13 tree13

A=size-shy‐property13 B=Phys-shy‐en-ty13

DIMENSION-shy‐OF13 (height13 tree)13

Stack13 of13 plates13 A=arrangement13 B=plurality13

GROUPING-shy‐OF13 (stackplates)13 B13 ALIGNED-shy‐IN-shy‐DIRECTION13 A13

Using13 VigNet13 and13 future13 work13

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 Sentence13 meaning13 diagram13

bull  John13 walked13 from13 the13 road13 to13 the13 old13 house13 with13 the13 red13 front13 door13

Seman4c13 Nodes13

Basic13 Approach13 ndash13 Seman-c13 types13 and13 rela-ons13 10013

Seman4c13 Rela4ons13

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Example13 create13 dependency13 structure13

10113

Input13 The13 truck13 chased13 the13 man13 down13 the13 long13 road13

Parse13 tree13 Dependency13 rela4ons13

Using13 VigNet13 and13 future13 work13

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

FrameNet13 for13 text-shy‐to-shy‐scene13 bull  Frames13 intui-vely13 provides13 correct13 level13 of13 abstrac-on13 However13 13 1  Text13 to13 scene13 (and13 other13 applica-ons)13 need13 more13 fine-shy‐

grained13 sense13 dis-nc-ons13 FrameNet13 frames13 are13 not13 graphically13 mo-vated13

2  FrameNet13 frames13 do13 not13 have13 internal13 structure13 13 no13 grounding13 13

3  Shallow13 seman-cs13 No13 obvious13 way13 of13 instan-a-ng13 frames13 to13 formally13 represent13 the13 meaning13 of13 a13 text13 passage13

4  FrameNet13 only13 records13 lexical13 and13 shallow13 conceptual13 knowledge13 How13 can13 we13 assert13 world-shy‐knowledge13 13

FrameNet13 and13 its13 limita-ons13 10213

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Selec-onal13 Restric-ons13 contd13

Grounded13 seman-cs13 through13 vignejes13 10313

Apple(self)13

Fruit(self)13 Shape_of(figureself13 shapespherical)13 Size_of(figureself13 sizesmall)13

Wash_small_fruit13 (washer13 theme13 sink)13

Front-shy‐Of(figure13 washer13 figure13 sink)13 Facing(figurewasher13 figuresink)13 Grasp(grasperwasher13 themetheme)13 Reach(reacherwasher13 targetsink)13

theme13 =13 x13 x13 =13 Fruit13 Size_of(figurex13 sizesmall)13 sink13 =13 Sink13 Washer13 =13 Sen-ent13

bull  Abbrevia-on13 Frame(selfx)13 13 13 egrave x=Frame13 bull  Allow13 use13 of13 variables13

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413

Frame-shy‐to-shy‐Frame13 Rela-ons13 contd13

ndash Network13 of13 directed13 rela-ons13 between13 frames13 bull  Causa-ve_of13 bull  Inchoa-ve_of13 bull  Perspec-ve_on13

Detaching13

Being13 detached13

Becoming13 detached13

Causa-ve_of13

Inchoa-ve_of13

Commerce_goods-shy‐transfer13

Commerce_buy13 Commerce_sell13

Perspec-ve_on13 Perspec-ve_on13

FrameNet13 and13 its13 limita-ons13 10413