A Unified Architecture

Stefanos Poulis Department of Mathematics

UCSD

A Unified Architecture for

Natural Language Processing (Deep Networks with Multitask Learning)

Motivation

  Natural Language Processing (NLP): Area, where “good” feature engineering is key to classifier performance

  Shallow classifier approach: Use your “good” features and feed them to your favorite shallow classifier (e.g. SVM)

  You will have “good” features only if you are a good “designer”   The above is dramatized in NLP tasks, where Multitask Learning is

often needed

  Multitask Learning: Learn features from one task to use them as features for another task

.

NLP tasks   Part-Of-Speech Tagging (POS): syntactic roles (noun,

adverb…)   Chunking: syntactic constituents (noun phrase, verb

phrase…)   Name Entity Recognition (NER) :person/company/

location…   Semantic Role Labeling: [John]ARG0 [ate]REL [the

apple ]ARG1 [in the garden]ARGM-LOC.

Features in NLP-Usual Approach

• Feed to a shallow classifier

Multitask Learning in NLP   Cascading Features: The most obvious way to achieve

MTL in NLP   Train one task, use this task as feature for another task.

 Eg: SRL: POS Parse tree Hand made features Shallow classifier (eg. SVM)

Multitask Learning in NLP   Shallow joint training:

  Hand-craft features, train on all tasks at the same time: Conditional Random Fields and statistical parsing models for POS, NER, chunking and relation extraction

o  Problem: joint labeling requirement   Learn independently by using different training sets but leverage

predictions jointly o  Problem: Not using the shared tasks at training

  SRL by joint inference and semantic parsing o  Problem: Not state of the art

Multitask Learning in NLP (Cont.)   Cascading Features: Good…? Bad?

Multitask Learning in NLP (Cont.)   Cascading Features: Good…? Bad?

  Questionable from machine learning point of view: Error propagation, extract hand made feature for each task

  How about an END-TO-END system that learns features completely implicitly?

  Yes, if it could achieve the performance of a shallow classifier

Neural Network Approach

Feature space 1

Feature space 2

  Craft a Deep Architecture.

  Train everything by back-propagation. (END-TO-END)   Architecture can unify all NLP tasks.

Neural Network Approach

Sharing part

  Train all tasks jointly.

  Can gain performance on all tasks.

The Deep Learning way: Implementation

Mapping of word of interest

 Feature vectors of size defined a priori

Concatenate feature vectors and feed to NN

  For SRL, want to tag a word w.r.t a verb, hence must see the full sentences at the same time.

The Deep Learning way: Generalization with Convolutional NN

Convolve by

•  Taking overlapping windows of size k

•  Concatenating

  Apply look up tables ,where W a matrix of parameters to be learnt

  An input sequence of words in the dictionary D, is transformed into a sequence of vectors

  Pre-processing, eg. stemming, lower case, capitalizations as a separate feature (YES/NO)

Word Mapping-1st Layer

€

s

€

s1,...,sn

€

n

€

Ws1...Wsn

  Embed each word into a d-dimensional space

  Decompose a word into K elements (features).   Represent words as tuples:

 For each word, i = {i1,…, iK} in {D1,…, DK}, where Dk is the dictionary for the kth- element, associate a lookup table LTW with parameters Wk

 Embed each word in a d-dimensional space by concatenating LTW’s

  Classifying with respect to a predicate:   In SRL, the label of a word depends on a predicate  Add a feature that encodes the relative distance of a word

w.r.t a predicate  For ith word, if the predicate is at posp use LTdist (i-posp)

Word Mapping-1st Layer (Cont.)

Word Mapping-1st Layer (Cont.)

Removing Time Dimension   Normal NN’s are NOT able to handle sentences of variable

length   Could use a window around the word to be labeled

 Might work for simple tasks (eg. POS)   Fails on more complex tasks eg.(SRL, role of a word depends on

a word far away from the sentence-outside of window)

  Solution: TDNN’s, time refers to order of words in the sentence

  Read the sentence in an online fashion: given a word sequence perform a convolution where Lj ’s are the parameters of the layer

The Deep Learning way: Generalization with Convolutional NN

•  Have extracted local features •  Shared weights through

time-space •  Still need to take care of time, since

local features depend on number of words

•  Take all windows of words in the sentence to predict the label of one word of interest

The Deep Learning way: Capture the most Relevant Features

•  Add layer that captures the most relevant features over the sentence

•  Feed the TDNN layer into a “Max” Layer, which takes the maximum over time

•  “Max” Layer is of fixed dimensions, thus next layers like those of regular NN’s

Deeper Architecture (cont.)   A TDNN layer performs a linear operator over the

input sequence.   Linear approach works for POS-NER but NOT for

SRL.   Can add one or more layers to the NN.   Compute output by where the matrix of

parameters is trained by back-propagation

€

ol = tanh Ll ⋅ ol−1( )

Deeper Architecture (cont.)   The size of the last output is the number of classes

considered in the NLP task

  A softmax layer follows, to ensure positivity and summation to 1

  Probabilities for each class of the NLP task are given

  The whole network is trained by the cross-entropy criterion

Deeper Architecture (Cont.)

Deep Joint Training   Related tasks, means that features for one task might be

useful for features in other tasks.   Deepest layer learns features implicitly for each work in D.   Reasonable to expect that training NNs on related tasks that

share deep layers, would improve performance.   Training the NN by looping over the tasks:

1.  Select the next task 2.  Select training example for this task at random 3.  Update the NN by taking a gradient step w.r.t this example 4.  Go to 1

  Labeling in NLP is expensive   Unlabeled data are abundant   Can leverage on unlabeled data by unsupervised learning and

jointly train supervised and unsupervised tasks   Language model: Discriminate a two-class classification task

  Word in the middle of the input window related to context/NOT

  Used ranking cost, sampled online

Leveraging Unlabeled Data

Results   Language model: was trained on a huge database, thus was

trained alone   Embedding obtained in the look-up table was very successful

Experiments   Used sections of PropBank dataset (labeled community dataset) for

training and testing SRL tasks   POS, NER and chunking, were trained with the window version ksz =5.   POS/NER linear models, chunking hidden layer of 200 units   Language model was trained with ksz =5 and 100 hidden units.

  Used two look-up tables: one of dimension wsz and one of dimension 2 (Capital- Not Capital)

  SRL NN had a convolution layer with ksz =5 and 100 hidden units followed by another layer of 100 units.   Used three look-up tables: one for the word, two for relative distances

  Language model was trained on Wikipedia, 631M- words embedded to “synonyms”.   Language model   SRL through Multitasking

MTL by Sharing only Look-up Tables

Sharing LT’s

•  Everything is trained by back-propagation

Results   SRL: good results when NOT trained alone   SRL + Language model by semi-supervised training: best

results

Results   Beat state-of-the-art, 14.30% vs 16.54%   250 times faster than state-of-the-art, .01 seqs to label a WSJ

sentence

Summary   General deep NN architecture for NLP

 Avoid hand-craft engineering   Extremely fast  MTL is advantageous  When leveraging on semi-supervised learning, results are superior

  Critics from NLP/Computational Linguistics community   Syntactic features are pivotal in learning  NN’s are hard to handle  NN are old (1997) !!

Summary   General deep NN architecture for NLP

 Avoid hand-craft engineering   Extremely fast  MTL is advantageous  When leveraging on semi-supervised learning, results are superior

  Critics from NLP/Computational Linguistics community   Syntactic features are pivotal in learning  NN’s are hard to handle  NN are old (1997) !!

  If training had started in 1997 when NN’s with vintage computers, it would finish at about NOW!

THANK YOU!