Neural models in NLPNatural Language Processing: Lecture 4
28.09.2017
Kairit Sirts
The goal of today’s lecture
• Explain word embeddings
• Explain the recurrent neural models used in NLP
2
Log-linear language model
y – the next word to predict
x – the context sequence: words, annotations etc
v – model parameters
f(x, y) – feature vector for the input-output pair (x, y)
3
The problem with log-linear models
Feature engineering
• Developing feature templates
• Which features are relevant to which problems?
• Experiment with subsets of features
• Features can be very complex
4
What if we could let the model learn the relevant features automatically?
Neural networks
5
1-hot representation
the girl with flowers is cute are were flower … …
The 1 0 0 0 0 0 0 0 0 0 0
girl 0 1 0 0 0 0 0 0 0 0 0
with 0 0 1 0 0 0 0 0 0 0 0
the 1 0 0 0 0 0 0 0 0 0 0
flowers 0 0 0 1 0 0 0 0 0 0 0
is 0 0 0 0 1 0 0 0 0 0 0
cute 0 0 0 0 0 1 0 0 0 0 0
… … … … … … … … … … … …
flower 0 0 0 0 0 0 0 0 1 0 0
6
What is the similarity between vectors for flower and flowers?
flowers 0 0 0 1 0 0 0 0 0 0 0
flower 0 0 0 0 0 0 0 0 1 0 0
7
the girl with flowers is cute are were flower … …
Features as distributed representations
8Deep Learning: What is meant by a distributed representation? https://www.quora.com/Deep-Learning-What-is-meant-by-a-distributed-representation/answer/Rangan-Majumder
Distributed word representations
9
f1 f2 f3 f4
flower 6 3 0 4
flowers 1 7 2 8
What is the cosine similarity between flower and flowers now?
Learning distributed word representations
The girl with the flowers is cute.
She has the flowers in her hand.
I picked these flowers myself.
The girl with a flower is cute.
She has a flower in her hand.
I picked this flower myself.
10
with the
has the
picked the
is cute
in her
myself
flowerswith a
has a
picked a
is cute
in her
myself
flower
11http://metaoptimize.s3.amazonaws.com/cw-embeddings-ACL2010/embeddings-mostcommon.EMBEDDING_SIZE=50.png
12
Word2Vec
13Mikolov et al., 2013. Efficient Estimation of Word Representations in Vector Space
CBOW – continuous bag of words
• w(t-2), w(t-1), w(t+1), w(t+2) – one-hot vectors
• – a row in the parameter matrix
• C – the set of context vectors
• c – the size of the context window
• - linear projection• d – embedding size
• 14
Skip-gram model
• Predict the context words
• w(t) – one-hot vector
• Maximize: z
Training word embeddings
• General principle – maximize the probability of the:• Middle word, given the context words (CBOW)
• Context words, given the middle word (skip-gram)
• In case of skip-gram:• Given T training words in context
• Maximize:
• Minimize:
16
Training word embeddings
• Typically trained with gradient descent• You will learn more sophisticated methods in other courses
• Initialize the parameter vectors/matrices (somehow)
• Repeat until convergence:
• - the set of all trainable parameters
• - learning rate
17
Softmax vs log-linear model
Softmax is a log-linear model
Log-linear:
Softmax:
18
The gradient of a log-linear model
19
Empirical count Expected count
The gradients in skip-gram model
20
c – context wordw – middle word
The problem with softmax gradients
• Computing is computationally very expensive. Why?
• The gradients always include the sum over the whole vocabulary
• This makes computation very inefficient
21
Negative sampling
The general idea:
Maximize the probability of the (word, context) pairs that came from the training data (instead of the probability of the context given the word)
Previously: maximize
Now: maximize
22
Skip-gram objective with negative sampling
Maximize:
• - the set of random negative samples
• In practice, the number of negative samples per each positive sample is between 2-20
23
Tools for training word embeddings
• Word2vec – Gensim includes both CBOW and skip-gram implementations
• Glove – optimizes the predictions of co-occurrence counts between words
• Polyglot
• Dependency-based word embeddings
24
Further reading on word embeddings
• Mikolov et al., 2013. Distributed representations of words and phrases and their compositionality
• Mikolov et al., 2013. Efficient estimation of word representations in vector space
• Goldberg and Levy, 2014. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method
• Pennington et al., 2014. GloVe: Global Vectors for Word Representation
• Al-Rfou’ et al., 2013. Polyglot: Distributed Word Representations for Multilingual NLP
• Levy and Goldberg, 2014. Dependency-based word embeddings
25
Regularities between word embeddings
26Vector Representations of Words: https://www.tensorflow.org/tutorials/word2vec
Word embedding models as neural networks
27CS231n Convolutional Neural Networks for Visual Recognition: http://cs231n.github.io/assets/nn1/neural_net.jpeg
One-hot vector of the input word
Prediction of the contextword (softmax) or whetherthe (context, word) pairbelongs to the Data (negative sampling)
The row corresponding To the input word in
Word embeddings
Recurrent Neural Networks
28http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/
RNN Language Model
29https://www.linkedin.com/pulse/what-i-learned-from-deep-learning-summer-school-2016-hamid-palangi
Machine Translation with RNN
30
http://cs224d.stanford.edu/lectures/CS224d-Lecture8.pdf
RNN music generation
31Music Language Modeling with Recurrent Neural Networks: http://yoavz.com/music_rnn/
Sequence Models
32The Unreasonable Effectiveness of Recurrent Neural Models: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Recurrent Neural Networks
33
- Initial state
- a nonlinear function
<s>and so on …
Non-linear activation functions
34http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/
Cross-entropy loss function
35https://theneuralperspective.com/2016/10/02/02-logistic-regression/
Training neural networks
36
• Typically with stochastic or mini-batch gradient descent• (full batch) GD – gradients are computed based on all training items
• Mini-batch GD – at each step compute the gradients based on a small number (a mini-batch) of training samples: for instance 20 or 32 or 128 etc
• Stochastic GD – gradients are computed based on a single training item
• Gradients are computed using back-propagation• BP is an algorithm for an efficient application of the chain rule
• There are several versions of gradient descent that set the learning rates in a clever way• RMSProp, AdaGrad, AdaDelta, Momentum, Adam
Gated units
• RNN-s are supposed to remember long contexts but in practice they don’t
• Gated units, such as LSTM or GRU include gates that control:• How much from the next input is read in
• How much from the previous hidden state is remembered or forgotten
• How much from the cell state is used in the output
37Figure 12 from Herath et al., 2016. Going Deeper into Action Recognition: A Survey.
Tools for creating and training neural networks
38
• Python libraries that perform symbolic gradient computation• Keras
• Tensorflow
• Theano
• PyTorch
• Dynet
• …
• The field is developing rapidly
RNN LM and word embeddings
39
• Inputs x – one-hot vectors
• Parameter matrix - word embeddings
• Training embeddings with word2vec or a similar model is faster than with RNNLM
• Pretrained word embeddings can be used to initialise the U matrix in RNNLM• Transfer learning
Further reading
• Understanding LSTM networks
• Mikolov et al., 2013. Linguistic Regularities in Continuous Space Word Representations
40
Recap
• Word embeddings are dense distributed representations of words
• Word embeddings are trained from (word, context) pairs using neural models
• Word embeddings can be viewed as automatically learned feature vectors
• Recurrent neural networks are neural sequence models often used in NLP
• Pretrained word embeddings can be used to initialize the embedding layer of the recurrent neural models with textual input
41