Qualitative Spatial Reasoning in Interpreting Text and Narrative

Ernest Davis

New York University

COSIT 2011

Thesis

Simple natural language texts and narratives often raise problems in commonsense spatial knowledge and reasoning of surprising logical complexity and geometric richness.

Outline

Examples: 5 literary texts, 6 (or 9) contrived texts– Fitting things into containers– Blocking, pursuing, escaping, and hiding– Household objects

Summary

Morals for research

Let Stalk Strine Afferbeck Lauder

Usage of “aorta”

“Aorta have more busses. An aorta mikem smaller so they don’t take up half the road. An aorta put more seats innem so you doan tefter stann all the time. An aorta have more room innem --- you carn tardly move innem air so crowded. Aorta do something about it.”

Unusual logical form

Forward inference to notice contradiction.

Infer: The speaker is a fool.

(Common in actual narratives, but unusual in inference systems).

Simple representation

AreaOf(Bus) =

NumberOfSeatsIn(Bus) * AreaOf(Seat)

Infer:

Area(Seat)↑ ∧ NumberOfSeatsIn(Bus)↑ ⇒ Area(Bus)↑

More general and deeper argument

The region occupied by a bus is (to simplify) the union of the regions occupied by the seats.

Two seats on a bus occupy disjoint regions.

The area of a collection of disjoint regions is the sum of the area of each region.

Infer

If

• each seat in bus b1 corresponds to a seat in bus b2;

• each seat in b1 is smaller than the corresponding seat in b2; and

• there are more seats in b2 than b1

then b2 must be bigger than b1.

Expressive Representation. Given:

Bus(b) ⇒

PlaceOf(b) = ∪sSeatsOf(b)PlaceOf(s).

Bus(b) ∧ s1≠ s2 ∧ s1,s2 SeatsOf(b) ⇒ DR(PlaceOf(s1),PlaceOf(s2)).

CollectionOfRegions(c) ∧ [∀r1,r2c r1 ≠ r2 ⇒ DR(r1,r2)] ⇒

AreaOf(∪r c r) = ∑ r c AreaOf(r).

Infer:

Bus(b1) ∧ Bus(b2) ∧ Injection(m,SeatsOf(b1),SeatsOf(b2)) ∧ [∀sSeatsOf(b1) AreaOf(PlaceOf(s)) <

AreaOf(PlaceOf(Apply(m,s)))]

⇒AreaOf(PlaceOf(b2)) >

AreaOf(PlaceOf(b1)).

Disclaimer

The logic is just a convenient notation.

I am not saying that a reasoning program has to use a logical representation.

I am saying that the reasoning here will pretty much have to carry out this inference.

Infer:Bus(b1) ∧ Bus(b2) ∧

Injection(m,SeatsOf(b1),SeatsOf(b2)) ∧ [∀sSeatsOf(b1) AreaOf(PlaceOf(s)) <

AreaOf(PlaceOf(Apply(m,s)))] ⇒AreaOf(PlaceOf(b2)) > AreaOf(PlaceOf(b1)).Representational issues:Sets of regions.Interaction of measure (Area) and

mereology (DR, Union).

The Winograd Schema ChallengeHector Levesque

Commonsense 2011

Winograd schema: A pair of sentences, differing in one or two words, with an ambiguity that is resolved oppositely.

Winograd (1972):

“The city councilmen refused the demonstrators a permit because they feared violence”

vs.“… because they advocated violence.”

Winograd Schema Challenge

Collecting corpus of schemas that• are effortlessly disambiguated by human

readers• sound natural.• can’t be solved using selectional

restrictions• are not easily Googlable.Challenge for AI. Less far-reaching than the

Turing Test, but less problematic.

The Winograd Schema ChallengeHector Levesque

Commonsense 2011

“The trophy does not fit in the suitcase because it is too large.”

“The trophy does not fit in the suitcase because it is too small.”

Gloss “The trophy is too large to fit’’ as “The trophy does not fit and no larger object fits, but some smaller object fits’’

Rule out incorrect interpretation for disambiguation

Incorrect reading: The suitcase is too large.

∼FitsIn(Trophy,Suitcase) ∧[∀s1s1⊃ Suitcase ⇒∼FitsIn(Trophy,s1)] ∧

[∃s2 s2⊂Suitcase ∧ FitsIn(Trophy,s2)].

Infer that this is impossible.

Representational issues

∼FitsIn(Trophy,Suitcase) ∧[∀s1s1⊃ Suitcase ⇒∼FitsIn(Trophy,s1)]

∧[∃s2 s2⊂Suitcase ∧ FitsIn(Trophy,s2)].

Interaction of subset with fit.

Existential quantification over regions.

Books in boxes

“Each of my books fits in this box, but they do not all fit in the box. If I had more boxes like this, I could fit them all.”

∀a,cArrangement(a,c) ≡

[∀xcFeasibleShape(PlaceOf(x,a),x)] ∧

[∀x1,x2cx1≠x2 ⇒

DR(PlaceOf(x1,a),PlaceOf(x2,a))]

M=My books. X = the inside of the box.∀bM∃aArrangement(a,{b}) ∧ PlaceOf(b,a)⊂X.∼∃aArrangement(a, M) ∧

∀b M PlaceOf(b,a)⊂X

∃c,a [∀y c Congruent(y,X)] ∧ Arrangement(a,M) ∧ ∀bM ∃ y c PlaceOf(b,a)⊂y.

Representational issues

∃c,a [∀y c Congruent(y,X)] ∧

RigidArrangement(a,M) ∧ ∀bM ∃ y c PlaceOf(b,a)⊂y.

• Arrangement of a collection of objects.

• 3-level quantifier alternation.

(A natural formulation in terms of the actual boxes has a 5-level quantifier alternation.)

A Representational Challenge

“The equipment came out of the box, but now I can’t fit it into the box.”

A common experience.

Geometrically impossible.

Geometric aspects.

A reasoner that knows more about geometry than people might interpret “it” as the box.

Dog out of the Kitchen

“I tried to keep the dog out of the kitchen by putting a chair in the middle of the doorway, but it was too wide.”

What was too wide?Why in the middle?How did the dog get into the kitchen?Spatio-temporal reasoning.

The chair blocks the dog

∀h Fixed(h,Frame,PosF) ∧

Fixed(h,Chair,PosC) ∧ Feasible(h) ∧ DR(PlaceOf(Dog,Start(h)),Kitchen) ⇒ DR(PlaceOf(Dog,End(h)),Kitchen)

Feasible(h) ≡

∀x,y,s Solid(x) ∧ Solid(y) ∧ x≠y ∧ State(s,h) ⇒

DR(PlaceOf(x,s),PlaceOf(y,s))

No placement of the chair blocks the dog

∼[∃pc

∀h Fixed(h,Frame,PosF) ∧

Fixed(h,Chair,pc) ∧ Feasible(h) ∧

DR(PlaceOf(Dog,Start(h)),Kitchen) ⇒

DR(PlaceOf(Dog,End(h)),Kitchen)]

Central placement

I put the chair in the center of the doorframe, because I know that, for any chair, if any placement blocks the dog, then there is a central placement that blocks the dog.

Representational issues

Spatio-temporal reasoning.

Alternating quantification over histories and object positions.

The Tale of Benjamin BunnyBeatrix Potter

Spatial Inferences for Narrative Coherence

The cat does not initially see the rabbits, because they are behind her head.

The cat cannot see the rabbits under the basket.

No light can penetrate under the basket.

The rabbits cannot get out without lifting the basket, which would disturb the cat.

“A Man and his Dog” Thomas Mann

Many dogs, they say, are the death of a hare, a single dog cannot achieve it, even one much speedier and more enduring than Bashan. The hare can “double” and Bashan cannot --- and that is all there is to it. … The hare gives a quick, easy, almost malicious twitch at right angles to the course and Bashan shoots past from his rear … Before he can stop, turn around, and get going in the other direction, the hare has gained so much ground that it is out of sight.

Inferences

• Before the double, Bashan and the hare are running in the same direction.

• Bashan is close behind the hare (a second or two)

• The distance of the “double” is at least comparable to the width of Bashan, but not large enough that jumping forward that distance would allow the hare to escape.

• The dog is faster than the hare, so could catch it if the hare didn’t double.

• The hare can change velocity discontinuously.

• The dog has a limit on the rate of deceleration.

• A pack of dogs can catch the hare by surrounding it, so doubling does not help.

The reader can visualize the scenario.

Representational issues

• Differential games

• Properties of the second derivative.

Household ObjectsButtons and Cord

• No end of hard problems.

• Hard to separate physics from geometry

• The physics is often not easy.

Buttons

“I tried to put the button through the hole, but it was too [large/small].”

“I tried to put the button through the hole, but it had been sewn too tightly to the coat.”

“I forgot that the top button was fastened, so when I took off the coat, it tore off.”

* “I forgot that the top button was unfastened …”

David Copperfield,Charles Dickens

“[Peggotty] gave [my head] a good squeeze. I know it was a good squeeze because, being very plump, whenever she made any little exertion after she was dressed, some of the buttons on the back of her gown flew off. And I recollect two bursting to the opposite side of the parlour while she was hugging me.”

Inferences

• The gown is too tight.

• The burst buttons are behind Peggotty and in front of David.

• The gown will now gap in back until the buttons are fixed.

Difficulties

• Characterize the position of a fastened button – doable.

• Characterizing the motion involved in fastening a button – difficult. (“Too small” refers to the action, not to the static state.)

• Sufficient characterization of the physics of fabric (“The gown is too tight”) – difficult.

Cord

“The power cord on the laptop would not reach from the desk to the outlet, so I got an extension cord. Then my wife objected to having an electric cord across the center of the living room (‘Men!’), so I laid it around the edge of the room, and hid it behind the furniture, but of course I had to get a much longer cord.”

Note the disambiguation of “it”, and the time sequence of “laid” and “had to get.”

Representational Issues

Cord(c) ∧ FeasibleShape(r,c) ⇒ LengthOf(AxisOf(r)) = CordLength(c)∃p FeasibleShape(p,C1) ^

Through(p,LivingRoom) ^ At(End1(p),Table) ^ At(End2(p),Outlet)

• Reasoning about arc lengths• Quantifying over paths

Death of Bill SykesOliver Twist, Charles Dickens

[H]e set his foot against the stack of chimneys, fastened one end of the rope tightly and firmly around it, and with the other made a strong running noose … He could let himself down by the cord to within a less distance of the ground than his own height, and had his knife ready in his hand to cut it then and drop.

At the very instant when he brought the loop over his head previous to slipping it beneath his arm-pits … he lost his balance …

InferencesWhat is the cause of Sykes’ death?What would have happened if he had fallen• Before attaching the rope to the

chimneys?• Before slipping the rope over his head?• After getting the rope under his armpits?If the rope had been much shorter? 10 feet

longer?If things had gone right, what would have

been the final state of the rope?

Summary:Text as a source for qualitative reasoning problems

Advantages: • Natural, interesting • Undeniable qualitative information• Wide range of

– Spatial relations– Logical forms– Directions of inference

Less susceptible to “Use simulation!” and “Use ML!” than other applications

Disadvantages

• Finding or contriving texts takes work.

• Does not yield a well-defined class of problems.

• Some important spatial characteristics e.g. shape are entirely implicit.

• Hard to separate spatial from other issues

• Often hard to be sure of logical form

• Difficult to connect logical form to text

Morals for research

WARNING!The following should not be taken

as career advice, particularly if you are pursuing:

• A degree• Tenure• Funding (at least US)

Morals for research

• QSR research based on close reading of one or few texts.

• Look at small, logically complex, heterogeneous problems.

• We don’t know how to evaluate these, empirically or theoretically. Table the issue of evaluation.

More morals

• Study common, geometrically complex, relations e.g. Blocks(c,s,x,k)

The collection of objects c as placed in state s blocks object x from entering region k.

• Problem: These appear in many different variants.

• Problem: What inferences?• Study common higher-level operators:

“Too φ’’.

More morals

• Get help from linguists and cognitive psychologists.

• Perhaps attempt a taxonomy of the QSR problems in text.

Final remark

At this point, automated commonsense reasoning is high-risk, high-payoff, like fusion reactors or SETI.

More fun than either of those, though.