Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Effective Scaling of Long-term Memory for Reactive Rule-based
Agents
Nate Derbinsky, Ph.D. (ABD)
Computer Scientist
University of Michigan
Intelligent Agents
• Autonomous, Persistent • Observes and acts upon an environment
• Uses and learns knowledge
• Directs ac<vity towards achieving goals
25 October 2011 2
Example: Ground Robo<cs
25 October 2011 3
• Mul<ple terrains, other agents, weather
• Movement, obstruc<ons Environment
• Patrols, search-‐and-‐rescue, explora<on, experiments, …
• Terrain, topological rela<ons, traffic paOerns, …
Tasks
• Days – years • Autonomy and interac<on with other agents, handlers
Agent
A Common Problem
Agents need effec<ve access to diverse informa<on – Factual – Experien<al
Agents need to maintain real-‐<me reac<vity in dynamic environments
< 50 msec.
25 October 2011 6
Approach: RBS
Rules Combinatorial set of possible condi<ons
Working Memory Match <me scales with WM size
25 October 2011 7
y = 0.0213e0.0034x R² = 0.9974
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
100 200 300 400 500 600 700 800 900 1000
Retrieval Tim
e (m
sec)
Number of Objects
Long-‐Term Memory (LTM)
Class of mechanism to help agents cope with dynamic, par<ally-‐observable environments – Encodes experience – Stores internally – Supports retrieval
25 October 2011 10
An Interes<ng Dichotomy: Stored Context
Seman2c “knowing”
Episodic “remembering”
25 October 2011 11
Agents with LTM are func2onally enhanced across a variety of problems
The Problem LTM for Reac/ve Agents
Support… – incremental encoding and storage of experience – access to stored knowledge
Requirements – Reac<vity: decisions < 50msec. – Scalability: support large amounts of knowledge – Generality: effec<ve across a variety of tasks
25 October 2011 12
This Work
Development and evalua<on of two LTMs – Integra<on within the Soar cogni<ve architecture – Efficient algorithms and data structures – Formal analysis & empirical evalua<on
Claims – Effec<ve and efficient across a variety of tasks – Scale computa<onally to…
• Large amounts of knowledge • Long agent life<mes
25 October 2011 13
Cogni<ve Architecture
Specifica<on of those aspects of cogni<on that remain constant across the life<me of an agent – Memory systems of agent’s beliefs, goals, experience – Knowledge representa<on – Func<onal processes that lead to behavior – Learning mechanisms
Goal. Develop and understand intelligence across a diverse set of tasks and domains
25 October 2011 15
Research Focus
25 October 2011 16
Biological Plausibility
Psychological Plausibility
Agent Func2onality
Leabra
ACT-‐R CLARION EPIC
Companions ICARUS LIDA
Graphical Soar
Soar: Dis<nc<ve Characteris<cs [Laird, Newell, Rosenbloom 1987]
• Diverse processing and learning mechanisms that support general problem solving methods
• Efficiently brings to bear large amounts of knowledge
• Applied to many applica<on domains – Language, cogni<ve modeling, games, tac<cs, robo<cs, …
• Public distribu<on and documenta<on – Major opera<ng systems (Windows, OS X, Linux, iOS) – Many languages (C++, Java, Python)
25 October 2011 17
Soar: Comparison to RBS Processing
25 October 2011 18
Rule Memory
Working Memory
Select Rule (conflict resolu<on)
Input Output Match Rules To WM
Execute Rule Ac<ons
Soar: Comparison to RBS Processing
• Operators: basic unit of delibera<on – Explicitly represent current operator
• Rules contain knowledge that – Propose Operators: what is possible? – Evaluate Operators: what is preferred? – Apply Operator: modify working memory
• All rules that match fire in parallel
25 October 2011 19
Fire rules to Evaluate operators
Input Output Fire rules to
Apply selected operator
Output Fire rules to Propose operators
Decide
Soar: Architecture Focus on Memory
25 October 2011 20
Percep<on Ac<on
Decision Procedure
Working Memory
Procedural Memory
Episodic Memory
Seman<c Memory
Reconstruc<on
LT Memory
Soar: Memory Access
25 October 2011 21
The reac2vity of a Soar agent is the 2me required to make a decision, which includes accessing and modifying long-‐term memories
Working Memory
Procedural Memory
Input
Output
Cue
Cue Matching
Result
Soar: Memory Evalua<on
Metrics
Memory Usage
Max Decision Time
Task Performance
Domains
Linguis<cs
Mobile Robo<cs
Games 25 October 2011 22
Episodic Memory
Seman<c Memory
Seman<c Memory Func/onal Analysis
25 October 2011 23
• Ontology • 4.5K classes, 250K facts
SUMO
• Lexicon • 212K senses, 820K asser<ons
WordNet
• “Common Sense” • 500K concepts, 5M facts
Cyc
• Access to large KBs
• Retrieval bias as a reasoning heuris<c
Seman<c Memory Integra/on
Representa<on • Symbolic triples
Encoding • Deliberate
Cue Seman<cs • Feature subset
25 October 2011 24
Seman<c Memory Efficient Implementa/on – [ICCM ‘10; AAAI ‘11]
• Incremental inverted index • Sta<s<cal query op<miza<on • Heuris<c search
Mapping to Set-‐Valued Stores
• Computa<on takes O(1) <me, affects O(1) memories • Class includes f(useful historical proper<es)
Locally Efficient Bias Func<ons
25 October 2011 25
Seman<c Memory Empirical Evalua/on – [ICCM ‘10]
Synthe2c Scaling Study
25 October 2011 26
• Scaling parameter: k • Nodes = k!, Edges = [k+1]!
7: 5k 3MB
8: 40k 28MB
9: 360k 292MB
10: 3.6M 2GB
0
0.1
0.2
0.3
0.4
0 0.2 0.4 0.6 0.8 1
Retrieval Tim
e (m
sec)
Cue Selec2vity
3.6M
362K
40K
5K
Objects
Seman<c Memory Empirical Evalua/on – [ICCM ‘10]
Synthe2c Scaling Study
25 October 2011 27
• Scaling parameter: k • Nodes = k!, Edges = [k+1]!
7: 5k 3MB
8: 40k 28MB
9: 360k 292MB
10: 3.6M 2GB
0
0.1
0.2
0.3
0.4
0 0.2 0.4 0.6 0.8 1
Retrieval Tim
e (m
sec)
Cue Selec2vity
3.6M
362K
40K
5K
Objects
Seman<c Memory Empirical Evalua/on – [ICCM ‘10]
Synthe2c Scaling Study
25 October 2011 28
• Scaling parameter: k • Nodes = k!, Edges = [k+1]!
7: 5k 3MB
8: 40k 28MB
9: 360k 292MB
10: 3.6M 2GB
0
0.1
0.2
0.3
0.4
0.5
1 2 3 4 5 6 7 8 9 10 11
Retrieval Tim
e (m
sec)
Cue Constraints
3.6M
362K
40K
5K
Objects
Seman<c Memory Empirical Evalua/on – [ICCM ‘10]
Lexicon Queries: WordNet Experimental Setup • 10 random nouns • Full sense (7 feat’s) • 10 trials
Results ≤ 0.3 msec (σ=0.0108)
>100x faster >3x more data
[Douglass et al. 2009]
25 October 2011 29
Seman<c Memory Empirical Evalua/on – [AAAI ‘11]
Long-‐Term Memory Agent
25 October 2011 30
“run” (v)
?
…
Problem. Ambiguous Cues Hypothesis. Retrieval History is Useful
Applica2on. Word Sense Disambigua<on
Seman<c Memory Empirical Evalua/on – [AAAI ‘11]
Word Sense Disambigua2on Experimental Setup • Input: “word”, POS • Given: WordNet v3 • Correct sense(s)
a|er each aOempt Efficiency ≤ 1.34 msec
Task Performance (2 corpus exp.)
25 October 2011 31
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Random Lesk Lesk-‐S Sta2c Frequency
Recency Dynamic Frequency
Base-‐level Ac2va2on
SemCor
Senseval-‐2
Senseval-‐3
Seman<c Memory Empirical Evalua/on – [BRIMS ‘11]
Mobile Robo2cs • Incremental map learning • Naviga<on and planning
25 October 2011 32
0 0.02 0.04 0.06 0.08 0.1
0 600 1200 1800 2400 3000 3600
0 1 2 3 4 5
0 600 1200 1800 2400 3000 3600
0
1000
2000
3000
4000
0 600 1200 1800 2400 3000 3600
Average Working Memory size
Average Msec./Decisions >2,000x faster than reac2vity req.
Maximum Msec./Decisions Increased reac2vity!
Map in Seman<c Memory (< 1MB)
Map in Working Memory
Seman<c Memory Summary
25 October 2011 33
• More than an order of magnitude faster than reac<vity requirement in prac<ce
Reac<vity
• Synthe<c: millions of objects • WordNet: >820K objects
Scalability
• Useful in linguis<cs and robo<cs
Generality
Episodic Memory Humans
Long-‐term, contextualized store of specific events [Tulving 1983]
What you “remember” vs. what you “know”
25 October 2011 34
Working Memory
Episodic Memory Integra/on
25 October 2011 36
Representa<on • Episode: connected di-‐graph • Store: temporal sequence
Encoding • Automa<c
Cue Seman<cs • Par<al graph-‐match • Recency biased
Episodic Memory
Encoding
Storage
Episodic Memory Efficient Implementa/on – [ICCBR ’09]
25 October 2011 37
• Interval-‐based representa<on, encoding, search, and reconstruc<on
• Scale with state changes (discrete edge +/-‐)
Temporal Con<guity
• Temporally-‐global structure index • Scale with structural dis<nc<ons
Structural Regularity
Efficient Encoding & Storage Incremental Dynamic-‐Graph Index
25 October 2011 38
State
Mem
ory
Time
Efficient Retrieval Overview
Cue matching is a constrained form of subgraph isomorphism – Unify two rooted graphs with labeled edges
We u<lize 2-‐phase matching to avoid expensive search [Forbus, Gentner, Law 1995] – Surface: novel search algorithm (interval walk), discrimina<on network (DNF graph)
– Structure: standard heuris<cs (MCV, DFS)
25 October 2011 39
Efficient Retrieval Retrieval Algorithms
25 October 2011 40
Sa2sfac2on
Mem
ory
Time
Reconstruc2on via Interval Tree [Kriegel et al. 2000]
Efficient Retrieval Surface Match Data Structures
1. Interval Walk – Maintain interval endpoint sor<ng via b+-‐trees – On cue, add leaf pointers to <me keyed priority queue
• Pop as necessary to process or
25 October 2011 41
( , 2me)
( , 2me)
Efficient Retrieval Surface Match Data Structures
2. Incremental Episode Scoring via DNF Graph – Cue edges serve as minimal propaga<on direc<ves • Maps to DNF SAT: sat(n) := sat(n) ^ sat(par(n))
– On / , update clause(s), possibly recurse
25 October 2011 42
sat( ) := ( ^ ^ )
Efficient Retrieval Scaling – [ICCBR ’09]
Interval Walk O( |Δ| * Temporal Selec<vity )
Incremental Episode Scoring O( |Δ| * Structural Selec<vity )
25 October 2011 43
Episodic Memory Empirical Evalua/on
25 October 2011 44
Useful: 7 general capabili<es Efficient: >100 cues, <50 msec.
Scalable: >48 hours, ~150 bytes – 2.5 kb/episode
Games
Robo<cs
PDDL
WSD
Episodic Memory Summary
25 October 2011 45
• Faster than reac<vity requirement for many tasks/queries in prac<ce
Reac<vity
• Days of RT (millions of episodes)
Scalability
• Useful in games, robo<cs, planning, linguis<cs
Generality
LTM for Intelligent Agents Contribu/ons
• Integrated effec<ve and efficient seman<c and episodic memories with Soar
• Novel methods that scale to large amounts of knowledge and long agent life<mes
• Empirically evaluated on numerous tasks – Linguis<cs, robo<cs, games, planning – Desktop pla�orms, robo<cs hardware, (and mobile!)
25 October 2011 46
LTM for Generally Intelligent Agents Looking Forward
Future Direc2ons – Integra<ng context – Automa<c structure learning – Reasoning with mul<ple sources of knowledge
LTMs make possible today… – Robust decision-‐making
• Improves with explora<on and interac<on
– Human-‐agent interac<on • Complexity • Believability
25 October 2011 47