Embed Size (px)
Citation preview
On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)
Huicheng Zheng, Christophe Laurent and Grégoire Lefebvre
13th September 2006
Thanks to the MUSCLE Internal Fellowship (http://www.muscle-noe.org).
ICANN’06
2
Outline• Introduction• Minimization of the ASSOM objective
function• Fast-learning methods
– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments• Conclusions
3
Motivation of ASSOM
• Learning “invariance classes” with subspace learning and SOM [Kohonen. T., et al., 1997]– For example: spatial-translation invariance
rectangles
circles
triangles
……
4
Applications of ASSOM• Invariant feature formation
[Kohonen, T., et al., 1997]• Speech processing
[Hase, H., et al., 1996]• Texture segmentation
[Ruiz del Solar, J., 1998]• Image retrieval
[De Ridder, D., et al., 2000]• Image classification
[Zhang, B., et al., 1999]
5
ASSOM Modules Representing Subspaces
The module arrays in ASSOM
Rectangular topology
Hexagonal topology
1b 2b Hb
Q
x
1Tbx
2ˆ )(L jx
2Tbx Hbx
T
A module representing the subspace L(j)
c
i
j
ci
j
6
Competition and Adaptation
• Repeatedly:– Competition: The winner
– Adaptation: For the winner and the modules i in its neighborhood
– Orthonormalize the basis vectors
2ˆmaxarg )(L j
jc x
)(')()( ),( ih
ic
ih t bxpb
xx
xxIxp
)(ˆ)()(),(
T)()(
i
thtt ic
ic
L
N×N matrix:)(' i
hb
)(ihb )(i
cp
7
Transformation Invariance• Episodes correspond to signal subspaces.• Example:
– One episode, S, consists of 8 vectors. Each vector is translated in time with respect to the others.
8
Episode Learning
• Episode winner
• Adaptation: for each sample x(s) in the episode X={x(s), s S} – Rotate the basis vectors
– Orthonormalize the basis vectors
Ss
jsc j
2)(ˆmaxarg )(L
x
)(')()( )),(( ih
ic
ih ts bxpb
9
Deficiency of the Traditional Learning Rule
• Rotation operator pc(i)(x(s),t) is an N×N matrix.
– N: input vector dimension
• Approximately:NOP (number of operations) ∝ MN2
– M: subspace dimension
10
Efforts in the Literature
• Adaptive Subspace Map (ASM) [De Ridder, D., et al., 2000]:– Drop topological ordering– Perform a batch-mode updating with PCA– Essentially not ASSOM.
• Replace the basis updating rule [McGlinchey, S.J., Fyfe, C., 1998]– NOP ∝ M2N
11
Outline• Introduction• Minimization of the ASSOM objective
function• Fast-learning methods
– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments• Conclusions
12
Minimization of the ASSOMObjective Function
XXx
xd)(
)(
)(~
2
2
)( )(
Ps
shE
i Ss
ic
iL
)()( ˆ~ii LL
xxx where:
(projection error)
P(X): probability density function of X
Solution: Stochastic gradient descent:
)('2
T)()(
)(
)()()()( i
hSs
ic
ih
s
sstht b
x
xxIb
)(t : Learning rate function
13
Minimization of the ASSOMObjective Function
)(tWhen is small:
Ss
ic
Ss
ic
s
sstht
s
sstht 2
T)(
2
T)(
)(
)()()()(
)(
)()()()(
x
xxI
x
xxI
In practice, better stability has been observed by the modified form proposed in [Kohonen, T., et al., 1997]
Ss
ic
ic
ss
ssthtt
i )()(ˆ
)()()()()(
)(
T)()(
xx
xxIM
L
14
Minimization of the ASSOMObjective Function
• corresponds to a modified objective function:)()( ticM
XXx
xd)(
)(
)(ˆ )()(m P
s
shE
i Ss
ic
iL
• Solution to Em:
Ss
ic
ic
ss
ssthtt
i )()(ˆ
)()()()()(
)(
T)()(
xx
xxIB
L
• When is small:)(t
)()( )()( tt ic
ic BM
15
Outline
• Introduction
• Minimization of the ASSOM objective function
• Fast-learning methods– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments
• Conclusions
16
Insight on the Basis Vector Rotation
• Recall: traditional learning
)()(ˆ
)()()()()),((
)(
)()(
ss
ssthtts
i
Ti
ci
cxx
xxIxp
L
)(')()( )),(( ih
ic
ih ts bxpb
17
Insight on the Basis Vector Rotation
)(ihb
)(),()(,
)( stsihc
ih xb
)()(ˆ
)()()(),(
)(
)(')()(
,ss
sthtts
i
ih
Ti
cihc
xx
bx
L
)()(ˆ
)()()()(
)(
)(')()(')(
ss
sstht
i
ih
Ti
ci
hi
hxx
bxxbb
L
scalar
projection
• For fast computing, calculate first, then scale x(s) with to get
• NOP ∝ MN
• Referred to as FL-ASSOM (Fast-Learning ASSOM)
),()(, tsihc
Scalar
),()(, tsihc )(i
hb
18
Insight on the Basis Vector Rotation
)(ihb
)(' ihb
)(),()(,
)( stsihc
ih xb
)(sx
19
Outline
• Introduction
• Minimization of the ASSOM objective function
• Fast-learning methods– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments
• Conclusions
20
Batch-mode Fast Learning(BFL-ASSOM)
• Motivation: Re-use the previously calculated during module competition.
)()(ˆ
)()()(),(
)(
)('T)()(
,ss
sthtts
i
ihi
cihc
xx
bx
L
• In the basic ASSOM, L(i) keeps changing with receiving of each component vector x(s). has to be re-calculated for each x(s).
)(ˆ )( siLx
)(ˆ )( siLx
21
Batch-mode Rotation• Use the solution to the modified objective
function Em:
Ss
Ti
ci
css
ssthtt
i )()(ˆ
)()()()()(
)(
)()(
xx
xxIB
L
• Subspace remains the same for all the component vectors in the episode. We can now use calculated during module competition.
)(ˆ )( siLx
22
Batch-mode Fast Learning
Ss
ihc
ih sts )(),()(
,)( xb
)()(ˆ
)()()(),(
)(
)(')()(
,ss
sthtts
i
ih
Ti
cihc
xx
bx
L
where ),()(, tsihc is a scalar defined by:
• Correction is a linear combination of component vectors x(s) in the episode.
• For each episode, one orthonormalization of basis vectors is enough.
)(ihb
23
Outline
• Introduction
• Minimization of the ASSOM objective function
• Fast-learning methods– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments
• Conclusions
24
Experimental Demonstration• Emergence of translation-invariant filters
– Episodes are drawn from a colored noise image
– Vectors in episodes are subject to translation
white noise image colored noise image
• Example episode (magnified):
25
Resulted Filters
1b
2b
FL-ASSOM BFL-ASSOM
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
6
0 5 10 15 20 25 30(×10
3)
FL-ASSOM
BFL-ASSOM
e
t
Decrease of the average projection error e with learning step t:
26
Timing Results
Times given in seconds for 1,000 training steps.
M: subspace dimension
N: input vector dimension
VU: Vector Updating time
WL: Whole Learning time
27
Timing Results
0
200
400
600
800
1000
1200
50 100 200 400 N
VU(s)
ASSOM
FL-ASSOM
BFL-ASSOM
0
200
400
600
800
1000
1200
2 3 4 M
VU(s)
ASSOMFL-ASSOM
BFL-ASSOM
Change of vector updating time (VU) with input dimension N:
Change of vector updating time (VU) with subspace dimension M:
Vertical scales of FL-ASSOM and BFL-ASSOM have been magnified 10 times for clarity.
28
Outline
• Introduction
• Minimization of ASSOM objective function
• Fast-learning methods– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments
• Conclusions
29
Conclusions
• The basic ASSOM algorithm corresponds to a modified objective function.
• Updating of basis vectors in the basic ASSOM correponds to a scaling of the component vectors in the input episode.
• In batch-mode updating, the correction to the basis vectors is a linear combination of component vectors in the input episode.
• Basis learning can be dramatically boosted with the previous understandings.
30
References• De Ridder, D., et al., 2000: The adaptive subspace map for image
description and image database retrieval. SSPR&SPR 2000.
• Hase, H., et al., 1996: Speech signal processing using Adaptive Subspace SOM (ASSOM). Technical Report NC95-140, The Inst. of Electronics, Information and Communication Engineers, Tottori University, Koyama, Japan.
• Kohonen, T., et al., 1997: Self-Organized formation of various invariant-feature filters in the adaptive-subspace SOM. Neural Computation 9(6).
• McGlinchey, S. J., Fyfe, C., 1998: Fast formation of invariant feature maps. EUSIPCO’98.
• Ruiz del Solar, J., 1998: Texsom: texture segmentation using Self-Organizing Maps. Neurocomputing 21(1–3).
• Zhang, B., et al., 1999: Handwritten digit recognition by adaptive-subspace self-organizing map (ASSOM). IEEE Trans. on Neural Networks 10:4.
31
Thanks and questions?
