Contextual Vocabulary Acquisition:From Algorithm to Curriculum

William J. Rapaport

Department of Computer Science & Engineering,Department of Philosophy, Department of Linguistics,

and Center for Cognitive Science

rapaport@cse.buffalo.eduhttp://www.cse.buffalo.edu/~rapaport

Contextual Vocabulary Acquisition• Active, conscious acquisition of a meaning for a word,

as it occurs in a text, by reasoning from “context”• CVA = what you do when:

– You’re reading– You come to an unfamiliar word– It’s important for understanding the passage– No one’s around to ask– Dictionary doesn’t help

• No dictionary• Too lazy to look it up :-)• Word not in dictionary• Definition of no use

– Too hard (& you’d need to do CVA on the definition!)– Not relevant to the context

• So, you “figure out” a meaning for the word “from context”– “figure out” = infer (compute) a hypothesis about

what the word might mean in that text– “context” = ??

What Does ‘Brachet’ Mean?(From Malory’s Morte D’Arthur [page # in brackets])

1. There came a white hart running into the hall with a white brachet next to him, and thirty couples of black hounds came running after them. [66]

2. As the hart went by the sideboard,the white brachet bit him. [66]

3. The knight arose, took up the brachet androde away with the brachet. [66]

4. A lady came in and cried aloud to King Arthur,“Sire, the brachet is mine”. [66]

5. There was the white brachet which bayed at him fast. [72]

18. The hart lay dead; a brachet was biting on his throat,and other hounds came behind. [86]

What Is the “Context” for CVA?• “context” ≠ textual context

– surrounding words; “co-text” of word• “context” = wide context =

– “internalized” co-text …• ≈ reader’s interpretive mental model of textual “co-text”

– … “integrated” with reader’s prior knowledge…• “world” knowledge• language knowledge• previous hypotheses about word’s meaning• but not including external sources (dictionary, humans)

– … via belief revision• infer new beliefs from internalized co-text + prior knowledge• remove inconsistent beliefs

“Context” for CVA is in reader’s mind, not in the text

Prior Knowledge Text

PK1

PK2

PK3

PK4

Prior Knowledge Text

PK1

PK2

PK3

PK4

T1

Integrated KB Text

PK1

PK2

PK3

PK4

T1I(T1)

internalization

B-R Integrated KB Text

PK1

PK2

PK3

PK4

T1I(T1)

internalization

P5

inference

B-R Integrated KB Text

PK1

PK2

PK3

PK4

T1I(T1)

internalization

P5

inference

T2

I(T2)

P6

B-R Integrated KB Text

PK1

PK2

PK3

PK4

T1I(T1)

internalization

P5

inference

T2

I(T2)

P6

T3

I(T3)

B-R Integrated KB Text

PK1

PK2

PK3

PK4

T1I(T1)

internalization

P5

inference

T2

I(T2)

P6

T3

I(T3)

B-R Integrated KB(the reader’s mind)

Text

PK1

PK2

PK3

PK4

T1I(T1)

internalization

P5

inference

T2

I(T2)

P6

T3

I(T3)

P7

Note: All “contextual” reasoning is done in this “context”:

Meaning of “Meaning”

• “the meaning of a word” vs. “a meaning for a word”

– “the” single, correct meaning– “of ” meaning belongs to word

– “a” many possible meanings• depending on textual context, reader’s prior knowledge, etc.

– “for” reader hypothesizes meaningfrom “context”, & gives it to word

• “The meaning of things lies not in themselves but in our attitudes toward them.”

– Antoine de Saint-Exupéry, Wisdom of the Sands (1948)

• “Words don’t have meaning; they’re cues to meaning!”“Words might be better understood as operators, entities that operate directly on mental states in what can be formally understood as a dynamical system.”

– Jeffrey L. Elman, “On Words and Dinosaur Bones: Where Is Meaning?” (2007)

• “We cannot locate meaning in the text…; [figuring out meaning is an] active, dynamic process…, existing only in interactive behaviors of cultural, social, biological, and physical environment-systems.”

– William J. Clancey, “Scientific Antecedents of Situated Cognition” (forthcoming)

CVA & Vocabulary Instruction

• People do “incidental” (unconscious) CVA– Possibly best explanation of how we learn vocabulary

• Given # of words high-school grad knows (~45K),& # of years to learn them (~18) = ~2.5K words/year

• But only taught ~10% in 12 school years

• Students are taught “deliberate” (conscious) CVAin order to improve their vocabulary

Overview of CVA Project

• From Algorithm…1. Implemented computational theory of how to

figure out (compute) a meaning for an unfamiliar wordfrom “wide context”

• …to Curriculum2. Convert algorithms to an improved, teachable

curriculum

1. Computational CVA

• Implemented in SNePS(Shapiro 1979; Shapiro & Rapaport 1992)

– Intensional, propositional semantic-networkknowledge-representation, reasoning, & acting system

• “intensional”:– can represent fictional objects

• “propositional”:– can represent sentences in a text

• “semantic network:– labeled, directed graph with nodes linked by arcs– indexed by node:

» from any node, can describe rest of network

– Serves as model of the reader (“Cassie”)

1. Computational CVA (cont’d)

• KB: SNePS representation of reader’s prior knowledge

• I/P: SNePS representation of word in its co-text

• Processing (“simulates”/“models”/is?! reading):

– Uses logical inference, generalized inheritance, belief revisionto reason about text integrated with reader’s prior knowledge

– N & V definition algorithms deductively search this “belief-revised, integrated” KB (the wide context) for slot fillers for definition frame…

• O/P: Definition frame – slots (features): classes, structure, actions, properties, etc.– fillers (values): info gleaned from context (= integrated KB)

Cassie learns what “brachet” means:Background info about: harts, animals, King Arthur, etc.No info about: brachetsInput: formal-language (SNePS) version of simplified English

A hart runs into King Arthur’s hall.• In the story, B12 is a hart.• In the story, B13 is a hall.• In the story, B13 is King Arthur’s.• In the story, B12 runs into B13.

A white brachet is next to the hart.• In the story, B14 is a brachet.• In the story, B14 has the property “white”.• Therefore, brachets are physical objects. (deduced while reading; PK: Cassie believes that only physical objects have color)

--> (defineNoun "brachet")

Definition of brachet:

Class Inclusions: phys obj,

Possible Properties: white,

Possibly Similar Items:

animal, mammal, deer,

horse, pony, dog,I.e., a brachet is a physical object that can be white and that might be like an animal, mammal, deer, horse, pony, or dog

A hart runs into King Arthur’s hall.A white brachet is next to the hart.The brachet bites the hart’s buttock.

[PK: Only animals bite]

--> (defineNoun "brachet") Definition of brachet: Class Inclusions: animal, Possible Actions: bite buttock, Possible Properties: white, Possibly Similar Items: mammal, pony,

A hart runs into King Arthur’s hall.A white brachet is next to the hart.The brachet bites the hart’s buttock.The knight picks up the brachet.The knight carries the brachet.

[PK: Only small things can be picked up/carried]

--> (defineNoun "brachet") Definition of brachet: Class Inclusions: animal, Possible Actions: bite buttock, Possible Properties: small, white, Possibly Similar Items: mammal, pony,

A hart runs into King Arthur’s hall.A white brachet is next to the hart.The brachet bites the hart’s buttock.The knight picks up the brachet.The knight carries the brachet.The lady says that she wants the brachet.

[PK: Only valuable things are wanted]

--> (defineNoun "brachet") Definition of brachet: Class Inclusions: animal, Possible Actions: bite buttock, Possible Properties: valuable, small,

white, Possibly Similar Items: mammal, pony,

A hart runs into King Arthur’s hall.A white brachet is next to the hart.The brachet bites the hart’s buttock.The knight picks up the brachet.The knight carries the brachet.The lady says that she wants the brachet. The brachet bays at Sir Tor.

[PK: Only hunting dogs bay]

--> (defineNoun "brachet") Definition of brachet: Class Inclusions: hound, dog, Possible Actions: bite buttock, bay, hunt, Possible Properties: valuable, small, white,I.e. A brachet is a hound (a kind of dog) that can bite, bay, and hunt, and that may be valuable, small, and white.

General Comments

• Cassie’s behavior human protocols

• Cassie’s definition OED’s definition:= A brachet is “a kind of hound which hunts by scent”

Fragment of reader’s prior knowledge:m3 = In “real life”, white is a color Member(Lex(white),Lex(color),LIFE)m6 = In “real life”, harts are deer AKO(Lex(hart),Lex(deer),LIFE)m8 = In “real life”, deer are mammals AKO(Lex(deer),Lex(mammal),LIFE)m11 = In “real life”, halls are buildings AKO(Lex(hall),Lex(building),LIFE)m12 = In “real life”, b1 is named “King Arthur” Name(b1,”King Arthur”,LIFE)m14 = In “real life”, b1 is a king Isa(ISA,b1,Lex(king),LIFE)(etc.)

m16 = if v3 has property v2 & v2 is a color & v3 v1 then v1 is a class of physical objects all(x,y,z)

({Is1(z,y),Member1(y,lex(color)),Member1(z,x)} &=> {AKO1(x,lex(physical object))})

Reading the story:m17 = In the story, b2 is a hart ISA(b2,lex(hart),STORY)m24 = In the story, the hart runs into b3

Does(b2,into(b3,lex(run)),STORY)(b3 is King Arthur’s hall) – not shown(harts are deer) – not shown

A fragment of the entire network,showing the reader’s mental context consisting of:• prior knowledge, the story, & inferences.

The definition algorithm…• searches this entire network,• abstracts parts of it,• & produces a hypothesized meaning for ‘brachet’.

The Algorithms1. Generate initial hypothesis by

“syntactic manipulation”• Algebra: Solve an equation for unknown value X• Syntax: “Solve” a sentence for unknown word X

– “A white brachet (X) is next to the hart” X (a brachet) is something that is next to the hart and that can be white.

– I.e., “define” node X in terms of immediately connected nodes

2. Deductively search wide context to update hypothesis• I.e., “define” word X in terms of some (but not all) other connected nodes

• Return definition frame.

Noun Algorithm• Generate initial hypothesis by syntactic manipulation

• Then find or infer from wide context:– Basic-level class memberships (e.g., “dog”, rather than “animal”)

• else most-specific-level class memberships• else names of individuals

– Properties of Xs (else, of individual Xs) (e.g., size, color, …)

– Structure of Xs (else …) (part-whole, physical structure…)

– Acts that Xs perform (else …) or that can be done to/with Xs– Agents that do things to/with Xs– … or to whom things can be done with Xs– … or that own Xs– Possible synonyms, antonyms

Verb Algorithm• Generate initial hypothesis by syntactic manipulation• Then find or infer from wide context:

– Class membership (e.g., Conceptual Dependency)• What kind of act is X-ing (e.g., walking is a kind of moving)• What kinds of acts are X-ings (e.g., sauntering is a kind of

walking)– Properties/manners of X-ing (e.g., moving by foot, slow walking)

– Transitivity/subcategorization information• Return class membership of agent, object, indirect object, instrument

– Possible synonyms, antonyms– Causes & effects

• [Also: preliminary work on adjective/adverb algorithm]

Belief Revision

• To revise definitions of words used inconsistently with current meaning hypothesis

• SNeBR (ATMS; Martins & Shapiro 1988, Johnson 2006):

– If inference leads to a contradiction, then:

1. SNeBR asks user to remove culprit(s)

2. & automatically removes consequences inferred from culprit

Revision & Expansion• Removal & revision being automated via SNePSwD by ranking all propositions

with kn_cat:most intrinsic info re: language; fundamental background infocertain (“before” is transitive)

story info in text (“King Lot rode to town”)

life background info w/o variables or inference (“dogs are animals”)

story-comp info inferred from text (King Lot is a king, rode on a horse)

life-rule.1 everyday commonsense background info (BearsLiveYoung(x) Mammal(x))

life-rule.2 specialized background info (x smites y x kills y by hitting y)

leastcertain questionable already-revised life-rule.2; not part of input

Belief Revision: ‘smite’• Misunderstood word:

– Initially believe that ‘smite’ means:“kill by hitting”

• Read “King Lot smote down King Arthur”– Infer that King Arthur is dead

• Then read: “King Arthur drew his sword Excalibur”– Contradiction!– Weaken definition to: “hit and possibly kill”

• Then read more passages in which smiting ≠> killing– Hypothesize that ‘smite’ means “hit”

Belief Revision: ‘to dress’• Well-entrenched word…

– Believe ‘to dress’ means “to put clothes on”– Commonsense belief:

• Spears don’t wear clothing

• … used in new sense:– Read “King Claudius dressed his spear”

• Infer that spear wears clothing• Contradiction!• Modify definition to: “to put clothes on OR to do something else”

• Read “King Arthur dressed his troops before battle”– Infer that ‘dress’ means: “to put clothes on OR to prepare for battle”

• Eventually: Induce more general definition:– “to prepare” (for the day, for battle, for eating…)

A Computational Theory of CVA1. A word does not have a unique meaning.2. A word does not have a “correct” meaning.

a) Author’s intended meaning for word doesn’t need to be known by readerin order for reader to understand word in context

b) Even familiar/well-known words can acquire new meanings in new contexts.c) Neologisms are usually learned only from context

3. Every co-text can give some clue to a meaning for a word.• Generate initial hypothesis via syntactic/algebraic manipulation

4. But co-text must be integrated with reader’s prior knowledgea) Large co-text + large PK more cluesb) Lots of occurrences of word allow asymptotic approach to stable meaning hypothesis

a) CVA is computablea) CVA is “open-ended”, hypothesis generation.

a) CVA ≠ guess missing word (“cloze”); CVA ≠ word-sense disambiguation

b) Some words are easier to compute meanings for than others (N < V < Adj/Adv)1. CVA can improve general reading comprehension (through active reasoning)2. CVA can & should be taught in schools

2. From Algorithm to Curriculum• State of the art in classroom CVA:

– Mauser 1984: “context” = definition!

– Clarke & Nation 1980: a “strategy” (algorithm?):1. Determine part of speech of word2. Look at grammatical context

• Who does what to whom?

3. Look at surrounding textual context• Search for clues (as we do)

4. Guess the word; check your guess

CVA: From Algorithm to Curriculum

• “guess the word” = “then a miracle occurs”

• Surely, computer scientists can “be more explicit”!

• And so should teachers!

From Algorithm to Curriculum (cont’d)

• We have explicit, rule-based (symbolic) AI theory of CVA Teachable!

• Goal:– Not: teach people to “think like computers”– But: explicate computable & teachable methods

to hypothesize word meanings from context• AI as computational psychology:

– Devise computer programs that faithfully simulate(human) cognition

– Can tell us something about (human) mind• Joint work with Michael Kibby (UB Reading Clinic)

– We are teaching a machine, to see if what we learn in teaching it can help us teach students better

“Contextual Semantic Investigation” (CSI):A Curriculum Outline

1. Teacher models CSI2. Teacher models CSI with student participation3. Students model CSI with teacher assistance4. Students do CSI in small groups5. Students do CSI on their own

CSI: The Basic Algorithm

I. Become aware of word X& of need to understand X

II. Repeat:A. Generate hypothesis H about X’s meaningB. Test H until H is a plausible meaning for X

in the current “wide” context

IIB. Test H

1. Replace all occurrences of X in sentence by H

2. If Sentence (X := H) makes sense then proceed with reading else generate new H

IIA. Generate H1. Make an “intuitive” guess H

2. If H fails or you can’t guess, then do in any order:

1. if you have you read X before& if you (vaguely) recall its meaning, then test that earlier meaning

2. if you can generate a meaning from X’s morphology, then test that meaning

3. if you can make an “educated guess” (next slide), then test it

IIA2c: Make an “Educated Guess”

i. Re-read X’s sentence slowly & activelyii. Determine X’s part of speechiii. Summarize entire text so fariv. Activate your PK about the topicv. Make inferences from text + PKvi. Generate H based on all this

IIA3: If all previous steps fail,then do CVA

a) “Solve for X”

b) Search context for clues

c) Create H

IIA3a: Solve for X

i. Syntactically manipulate X’s sentenceso that X is the subject

ii. Generate a list of possible synonyms(as “hypotheses in waiting”)

IIA3b: Search context for clues

i. If X is a noun, then search context for clues about X’s…

• class membership• properties• structure• acts• agents• comparisons• contrasts

IIA3b: Search context for cluesii. If X is a verb,

then search context for clues about X’s…

• class membership what kind of act Xing is what kinds of acts are Xings

• properties of Xing (e.g., manner)

• transitivity look for agents and objects of Xing

• comparisons & contrasts

IIA3b: Search context for clues

iii. If X is an adjective or adverb, then search context for clues about X’s…

• class membership is it a color adjective, a size adjective, a shape adjective, etc.?

• contrasts is it an opposite or complement of something else mentioned?

• parallels is it one of several otherwise similar modifiers in the sentence?

IIA3c: Create H• Aristotelian definitions:

– What kind of thing is X?– How does it differ from other things of that kind?

• Schwartz & Raphael definition “map”– What is X?– What is it like?– What are some examples?

• Express (“important” parts of) definition framein a single sentence

Computation & Philosophy

• Computational philosophy =– Application of computational (i.e., algorithmic) solutions

to philosophical problems

• Philosophical computation =– Application of philosophy to CS problems

CVA as Philosophical Computation

• Origin of project:– Rapaport, “How to Make the World Fit Our Language” (1981)

• (Intensional) theory of a word’s meaning for a personas the set of contexts in which person has heard or seen word.

• Could that notion be made precise?• Semantic-network theory offered a computational tool• Developed into Karen Ehrlich’s CS Ph.D. dissertation (1995)

• Later, learned that computational linguists,reading educators, L2 educators, psychologists,…were all interested in this– A really interdisciplinary cognitive-science problem

CVA as Computational Philosophy

1. CVA & holistic semantic theories:– Semantic networks:

• “Meaning” of a node is its location in the entire network– Holism:

• Meaning of a word is its relationships to all other words in the language– Problems (Fodor & Lepore):

• No 2 people ever share a belief• No 2 people ever mean the same thing• No 1 person ever means the same thing at different times• No one can ever change his/her mind• Nothing can be contradicted• Nothing can be translated

– CVA offers principled way to restrict “entire network”to a useful subnetwork• That subnetwork can be shared across people, individuals, languages,… • Can also account for language/concept change

1. Via “dynamic”/“incremental” semantics

CVA as Computational Philosophy & Philosophical Computation (cont’d)

2. CVA and the Chinese Room– Searle’s CR argument from semantics:

1. Computer programs are purely syntactic2. Cognition is semantic3. Syntax alone does not suffice for semantics No purely syntactic computer program can exhibit semantic cognition

– How would Searle-in-the-Room figure out the meaning of an unknown squiggle?• By CVA techniques!

– “Syntactic Semantics” (Rapaport 1985ff)2. Syntax does suffice for the kind of semantics needed for NLU in the CR

1. All input—linguistic, perceptual, etc.—is encoded in a single network(or: in a single, real neural network: the brain!)

2. Relations—including semantic ones—among nodes of such a networkare manipulated syntactically

1. Hence computationally (CVA helps make this precise)

Summary• Contextual Vocabulary Acquisition project is:

– Computational philosophy• And computational psychology!

– Philosophical computation

– With applications to:• Computational linguistics• Reading education