Upload
phamtram
View
216
Download
2
Embed Size (px)
Citation preview
IST 4Information and Logic
mon tue wed thr fri30 M1 1= todayT
6 M1
13 1 2M2x= hw#x out
oh oh
oh oh
= todayT
M
20 227 M2
x= hw#x due
idt
oh oh
h h Students’ MQ27 M2
4 311 3 4Mx= MQx out
midtermsoh
oh oh
oh = office hours oh Students MQpresentations
11 3 418 4 525
Mx= MQx out
Mx= MQx dueoh oh
ohoh
25
1 5oh
oh
oh
oh
T
MQsMQs1. Everyone has a gift! (Tuesday)
2. Memory (Thursday)
Tuesday 6/2 2:30pm –
1. Christopher Haack: The gift of resilience
Tuesday, 6/2, 2:30pm
p g
2. Joon Lee: Settling is not an option
3 Spencer Strumwasser: The gift of dyslexia 3. Spencer Strumwasser: The gift of dyslexia
4. Richard Zhu: The gift of memory
5 A h i H i Th ift f i l iti5. Ashwin Hari: The gift of musical composition
6. Jessica Nassimi: Evolution—a gift in disguise
7. Serena Delgadillo: The gift of self-expression
8. Megan Keehan: Gift of motherliness
9. Zane Murphy: Grandmother and the piano
Thursday 6/4 2:30pm –Thursday, 6/4, 2:30pm
1. Connor Lee: Memory is a fickle thing – blessing or curse
2. Pallavi Aggarwal: The wonders of human memory
3. Peter Kundzicz and Anshul Ramachandran: Muscle memories
4. Siva Gangavarapu: A cultural retrospection. g p p
5. Philip Liu: The light of other days
6 Jason Simon: Math and Broadway6. Jason Simon: Math and Broadway
7. Yujie Xu: Memory v.s. ESL
8 C l Zh Wh 8. Celia Zhang: When memory sours
Last LectureGates and circuits AON: AND, OR, Not
LT: Linear ThresholdLT: Linear Threshold
>>
abc
ab >>
>>
abc
ab >
>
>
>
bc
abc
abc
>>>>
>>
>>
bc
abc
abc
Last LectureAON: AND, OR, Not LT: Linear Threshold
General construction for symmetric functions
AON 54
AONLT-l
*
*
2LT-nl *
Exponential gap in sizeWhat are the symmetric functions that can be computed functions that can be computed by a single LT gate? * = it is optimal
Linear Threshold and SYM
LT: Linear Threshold
Symmetric Functions and LT Circuits
Q: How is SYM related to LT1 ??
Q: Which class has more functions?
Definitions:
Q
Definitions:
(1) SYM = the class of Boolean symmetric functions
(2) LT1 = the class of Boolean functions that can be (2) LT1 the class of Boolean functions that can be realized by a single LT gate.
AND, OR, XOR and MAJ are symmetric functions
Q Whi h t i f ti i LT ?
|X| AND OR XOR MAJ
Q: Which symmetric functions are in LT1?
0 0 0 0 0
1 0 1 1 01 0 1 1 0
2 0 1 0 1
3 1 1 1 1
LT1 LT1 LT1not LT1
LT1 = the class of Boolean functions that can be realized by a single LT gate.
LT1 LT1 LT1not LT1
realized by a single LT gate.
Definition: A t i B l f ti i i TH if it h t t A symmetric Boolean function is in TH if it has at most a single transition in the symmetric function table
= a transition
|X| AND OR XOR MAJ
0 0 0 0 0
1 0 1 1 0
2 0 1 0 1
3 1 1 1 1
Not in THIn TH
The Class TH is in LT1
|X| TH0 TH1 TH2 TH3 TH0 TH1 TH2 TH3
0 1 0 0 0 0 1 1 10 1 0 0 0 0 1 1 11 1 1 0 0 0 0 1 1
1 1 1 0 0 0 0 12 1 1 1 0 0 0 0 1
3 1 1 1 1 0 0 0 0
Q: How is TH related to SYM and LT1 ??
We know that:
Q
We know that:
We Proved that:
SYM LT1TH
TH is exactly in the intersection of SYM and LT1
Theorem:
Q: What are the 4 functions?
Proof: Not today... you might want to try and prove it...
Q: What are the 4 functions?
SYM LT1TH
???
LT1 Function that is not SymmetricLT1 Function that is not Symmetric
0 00 1
-1-2
00
11
1 01 1
0-1
10
-1-1
lLinear Threshold Circuits
for symmetric functionsfor symmetric functions
5AON 54
AONLT-l
L l 2LT-nl
General construction for symmetric functions
|X| XOR Q: compute XOR with TH gates?
0 01 1
with TH gates?
2 0
1|X| TH1 TH2 TH1+TH2-1
0 0 1 01 1 1 12 1 0 02 1 0 0
LT Depth-2 Circuits
TH1
+-1
TH2 |X| TH1 TH2 TH1+TH2-1 |X| TH1 TH2
0 0 1 01 1 1 11 1 1 12 1 0 0
Generalization
|X| f( )
z
|X| f(x)0 0
1 1
2 1
3 0
4 04 0
Generalization
|X| f( )
z
|X| f(x)0 0
1 1
2 1
3 0
4 04 0
Generalization
|X| f( ) TH1
z
|X| f(x) TH1
0 0 0
1 1 1
2 1 1
3 0 1
4 0 14 0 1
Generalization
|X| f( ) TH1 TH3
z
|X| f(x) TH1 TH3
0 0 0 1
1 1 1 1
2 1 1 1
3 0 1 0
4 0 1 04 0 1 0
Generalization
|X| f( ) TH1 TH3 Σ 1
z
|X| f(x) TH1 TH3 Σ -10 0 0 1 0
1 1 1 1 1
2 1 1 1 1
3 0 1 0 0
4 0 1 0 04 0 1 0 0
|X| f( ) TH1 TH3 Σ 1|X| f(x) TH1 TH3 Σ -10 0 0 1 0
1 1 1 1 1
2 1 1 1 1
3 0 1 0 0
4 0 1 0 04 0 1 0 0
Generalization to EQz EQ
00
00 0
1
1
1
n
1
00
Generalization to EQz EQ
00 1
1
1
12
0
1
0 1
Generalization to SYMz M
+-1
Q:h h l fWhat is the generalization to arbitrary symmetric functions?
Generalization to SYMz M
Q:Q:What is the generalization to arbitrary symmetric functions?
A:A:Consider the symmetric function table, it is a sum of non-overlapping 1-intervals
0
0
1
1 Sum of two TH functions
Back to XORX
n TH gates for XOR of n variables
0 0
1
2
1
0
3
4
1
0
5 1
LT l Circuit Design Algorithm for SYMLT-l Circuit Design Algorithm for SYM
f(X)
0 1
f(X)
0
1
2
1
1
02
3
0
1Subtract 1 for every
4
5
1
0
Subtract 1 for everyisolated 1-block
67
11
The Layered Construction for SYM -Some History
Saburo Muroga1925- 2009
1959
Was born in Japan Majority Decision
PhD in 1958 from Tokyo U, Japan
1960-1964: Researcher at IBM Research, NY,
1964-2002: professor at the University of Illinois, Urbana-Champaign
Saburo Muroga1925- 2009
HW#5 problem 2a
neural circuits and logicg
some more historysome more history...
Being Homeless and Interdisciplinary Research
W M C ll hWarren McCulloch1899 - 1969
Walter Pitts1923 - 1969
Neurophysiologist, MD Logician, Autodidactg ,
Warren McCulloch arrived in early 1942 to the University of Chicago, invited Pitts, who was homeless, to live with his family
h ll h d P ll b d In the evenings McCulloch and Pitts collaborated. Pitts was familiar with the work of Leibniz on computing.They considered the question of whether the nervous system is a kind of universal computing device as described by Leibnizuniversal computing device as described by Leibniz
This led to their 1943 seminal neural networks paper:A Logical Calculus of Ideas Immanent in Nervous Activity
Warren McCulloch W l Pi
ImpactWarren McCulloch
1899 - 1969 Walter Pitts1923 - 1969
Neurophysiologist, MD Logician, Autodidactg ,
This led to their 1943 seminal neural networks paper:p pA Logical Calculus of Ideas Immanent in Nervous Activity
Neural networks d Logic Ti MNeural networks and Logic Time Memory
Threshold Logic and Learning
State Machines
neural circuits and memorym m y
computing with dynamicscomputing with dynamics
Linear ThresholdSome AdjustmentsSome Adjustments
Linear Threshold (LT) gateLinear Threshold (LT) gate
-tthreshold
t
-t
-1
t
1
-1
AND Function with {0,1}
-21
-21
0 00 1
1 0
-2-11
0001 0
1 1-10
01
AND Function with {-1,1}
Th AND f ti f t i bl ith { 1 1} ???The AND function of two variables with {-1, 1}: ???
-1-1-11
-3-1--
1-111
11
1-11-+
AND Function with {-1,1}
Th AND f ti f t i bl ith { 1 1} ???The AND function of two variables with {-1, 1}: ???
-1-1-11
-3-1
1-111
11
1-11
Linear Threshold with MemoryEl h t b l f i d i A i ltElephants are symbols of wisdom in Asian culturesand are famed for their exceptional memory
A memory nosey
Remembers the last f(X)
Feedback NetworksE lExample
-1 -1 00
weights thresholds
Th t t f th t k th t th t d The state of the network: the vector that corresponds to the states (noses…) of the gates
Feedback NetworksExampleExample
1 2Label the gates
-1 -1 00 1 1
1 2Label the gates
-1 -1 00
Feedback NetworksExampleExample
1 2
-1 -1 00 1 1
1 2
1-1 -1 00
Feedback NetworksExampleExample
1 2
-1 -1 00 -1 1
1 2
1-1 -1 00
Feedback NetworksExampleExample
1 2
-1 -1 00 -1 1
1 2-1
-1 -1 00
11 i bl -11 is a stable state
Feedback NetworksExampleExample
1 2
-1 -1 00 1 1
1 2
-1 -1 00
Feedback NetworksExampleExample
1 2
-1 -1 00 1 1
1 21
-1 -1 00
Feedback NetworksExampleExample
1 2
-1 -1 00 1 -1
1 21
-1 -1 00
Feedback NetworksExampleExample
1 2
-1 -1 00 1 -1
1 2-1
-1 -1 00
1 1 i bl 1-1 is a stable state
Feedback NetworksExampleExample
1 2
-1 -1 00 1 1
1 2
-1 -1 00
State transition diagram (state space)1
The node
11 -11
2
1state
The node that computes -1-11-1
2Q:Is -1-1 a stable state?
Feedback NetworksExampleExample
1 2
-1 -1 00 -1 -1
1 2-1
-1 -1 00
Answer: NoAnswer: No
Q:Is -1-1 a stable state?
Feedback NetworksExampleExample
1 2
-1 -1 00 1 -1
1 2-1
-1 -1 00
11 111
11 -11
2-1-11-1
1
Feedback NetworksExampleExample
1 2
-1 -1 00 -1 -1
1 2-1
-1 -1 00
11 111
11 -11
2-1-11-1
1
Feedback NetworksExampleExample
1 2
-1 -1 00 -1 1
1 2-1
-1 -1 00
11 111
11 -11
2 2-1-11-1
1
Feedback NetworksExampleExample
1 2
-1 -1 00 -1 1
1 2
-1 -1 00
11 111stable states
11 -11
2 2-1-11-1
1
neural circuits and memorym m y
associative memoryassociative memory
Feedback NetworksComputing with DynamicsComputing with Dynamics
1stable states11 -11
2
1
2
stable states
-1-11-1
2
1
2
Input:initial state
Output:stable state
FeedbackNetwork
n t al state
11 -1111 11
Feedback NetworksComputing with DynamicsComputing with Dynamics
1stable states11 -11
2
1
2
stable states
-1-11-1
2
1
2
Input:initial state
Output:stable state
FeedbackNetwork
n t al state
11 1-111 1 1
Feedback NetworksComputing with DynamicsComputing with Dynamics
Input: Output:Associative Memory“The Leibniz-Boole Machine”p
initial statep
stable stateThe Leibniz Boole Machine
FeedbackNetwork
Feedback NetworksComputing with DynamicsComputing with Dynamics
Input: Output:Associative Memory“The Leibniz-Boole Machine”p
initial statep
stable stateThe Leibniz Boole Machine
FeedbackNetwork
Feedback NetworksComputing with DynamicsComputing with Dynamics
Input: Output:Associative Memory“The Leibniz-Boole Machine”p
initial statep
stable stateThe Leibniz Boole Machine
FeedbackNetwork
Feedback NetworksComputing with DynamicsComputing with Dynamics
Input: Output:Associative Memory“The Leibniz-Boole Machine”p
initial statep
stable stateThe Leibniz Boole Machine
FeedbackNetwork
Feedback NetworksComputing with DynamicsComputing with Dynamics
Input: Output:Associative Memory“The Leibniz-Boole Machine”p
initial statep
stable stateThe Leibniz Boole Machine
FeedbackNetwork
Feedback NetworksComputing with DynamicsComputing with Dynamics
Input: Output:Associative Memory“The Leibniz-Boole Machine”p
initial statep
stable stateThe Leibniz Boole Machine
FeedbackNetwork
Feedback NetworksComputing with DynamicsComputing with Dynamics
Input: Output:Associative Memory“The Leibniz-Boole Machine”p
initial statep
stable stateThe Leibniz Boole Machine
FeedbackNetwork
Feedback NetworksComputing with DynamicsComputing with Dynamics
Input: Output:Associative Memory“The Leibniz-Boole Machine”p
initial statep
stable stateThe Leibniz Boole Machine
FeedbackNetwork
????
Who is this person?????
John Hopfield
f ld d l ( l h )Feedback Networks
Hopfield Model (Caltech 1982)
Feedback NetworksJohn Hopfield
f ld d l ( l h )
1 2
Hopfield Model (Caltech 1982)
-1 -1 00 1 1
-11 20 0
1 1
1
i = node i 0 = threshold ti
-1
1 = state vi -1 = weight of edge (i,j)
The matrix descriptionm p
Feedback Networks/The Vector/Matrix Description
An n node feedback network can be specified by:An n node feedback network can be specified by:• W an nxn matrix of weights• T an n vector of thresholds f• V an n vector of states
1
4
2
34
5
The Matrix DescriptionE lExample
An n node feedback network can be specified by:An n node feedback network can be specified by:• W an nxn matrix of weights• T an n vector of thresholds f• V an n vector of states
1 20 0
-121
-1
11 1
-1
-1
The Matrix DescriptionComputationComputation
C t ti i N (W T)Computation in N= (W,T)1
4
2
34
5by column
Order of computationf mp
serial and parallel
Modes of OperationModes of Operation
Q: when do the nodes compute?Q: when do the nodes compute?
Serial mode: one node at a time (arbitrary order)
-1
( y )
21-1
Modes of OperationModes of Operation
Q: when do the nodes compute?Q: when do the nodes compute?
Serial mode: one node at a time (arbitrary order)
-1
( y )
21-1
Modes of OperationModes of Operation
Q: when do the nodes compute?Q: when do the nodes compute?
Serial mode: one node at a time (arbitrary order)
-1
( y )
21-1
Fully-Parallel mode: all nodes at the same time-1
21-1
Three examplesmp
Example 1Serial Mode – Symmetric Weight MatrixSerial Mode – Symmetric Weight Matrix
21-1
-11
The state space:
1stable states
The state space:
11 -11
2
1
2-1-11-1
1
Example 2Fully Parallel (FP) Mode Symmetric Weight MatrixFully-Parallel (FP) Mode – Symmetric Weight Matrix
Q: how does the state space look?
21-1
Q p
21-1
start with 11start with 11
It’s acycle!cycle!
Example 2Fully Parallel (FP) Mode Symmetric Weight MatrixFully-Parallel (FP) Mode – Symmetric Weight Matrix
21-1
21-1
h The state space:
stable states11 -11
l f l th 2-1-11-1
cycle of length 2
Example 2Fully Parallel Mode Symmetric Weight Matrix
3AntisymmetricFully-Parallel Mode – Symmetric Weight MatrixAntisymmetric
W T = −W21
-1
1
11W W
-1
Q: how does the state space look?p
Example 3Fully Parallel Mode Antisymmetric Weight Matrix
1
Fully-Parallel Mode – Antisymmetric Weight Matrix
21-1
Q: how does the state space look?
cycle of length 4
Example 3Fully Parallel Mode Antisymmetric Weight Matrix
1
Fully-Parallel Mode – Antisymmetric Weight Matrix
21-1
The state space:
11 -11
-1-11-1
cycle of length 4
The Three Cases
11 -1111 -11 11 -11
-1-11-1-1-11-1 -1-11-1
1 2 3
Wmode symmetric antisymmetricCycle lengths
serial 1 ?1Example #
fully-parallel 1,2 42 3
The Three Cases
Wmode symmetric antisymmetricCycle lengths
mode ymm ymm
serial 1 ?1
Example #fully-parallel 1,2 4
2 3
Example #
1 Hopfield 1982
2 G l 19852 Goles 1985
3 Goles 1986
Proof Ideas
C l l th W symmetric antisymmetricCycle lengths Wmode symmetric antisymmetric
serial 1 ?1Example #
fully-parallel 1,2 4
1
2 3The proofs of these three results use the conceptof an energy function
For the serial mode:
Sh th tShow that:
Namely, stable states are local max of the energy E
Questions on Convergence
Posted on the class web site
Cycle lengths Wd symmetric antisymmetric
Posted on the class web site
Cycle lengths Wmode symmetric antisymmetric
serial 1 ?1Example #
fully-parallel 1,2 42 3
1 Hopfield 1982
2 Goles 1985 Q1: Are the three cases “distinct”?2 Goles 1985
3 Goles 1986
Q
Q2: Elementary proof? (wo/energy)