NIPS2015 End-To-End Memory Networks

S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus

Preferred Infrastructure @unnonouno

2016/01/20 NIPS2015@

Memory networks

l 2013Facebook

l

lend-to-end

2

1

3

4

lChainer

l l 300

l Chainer

5

bAbI task

l

l 177

6

l l : {x1, x2, , xn} l : q l

l l : a

l l : A, B, C d x V l : W V x d l d: V:

7

8

l l F.sum(model.A(x), axis=1)

9

V

n

1 3 2 5 1x=

ID

=

l xiAmi

10

1 3 2 5 1x1=

4 3 1 7x2=

1 3 4 8 9x3=

m1m2m3m4

A

l Bu

11

B 3 4 1 7 9q =

u =

l miuSoftmaxpiAttention

l p = F.softmax(F.batch_matmul(m, u))

12

m1m2m3m4 u

p1p2p3p4 pi = softmax(miTu)

xiCci

13

1 3 2 5 1x1=

4 3 1 7x2=

1 3 4 8 9x3=

c1 c2 c3 c4

C

l cipio l o = F.batch_matmul(F.swapaxes(c ,2, 1), p)

14

p1p2p3p4

c1 c2 c3 c4x

=

o

l uoW

l loss = F.softmax_cross_entropy(model.W(u + o), a)

15

o u

+ W

lxiAmiCci lqBu l miusoftmaxpi

l cipio l o + uWa

softmax cross entropyloss

16

17

BoW

l

l l

18

19

l Adjacent l l Ak+t = Ck l B = A1

l piqx

l Layer-wise l A1 = A2 = l C1 = C2 =

20

temporal encoding

l

l

21

x1 = Sam walks into the kitchen x2 = Sam walks into the bedroom

q = Where is Sam?

20

22

3Adjacent

position encoding

l l

23

PE

24

lLinear start (LS) lsoftmax

l Random noise (RN) l10% l

25

26

l

l l

l

27

28

lend-to-end

l

l

29

lend-to-end

lattention

l6%

l

30