NLP Applications using Deep Learning - GitHub Pages · 2018. 4. 29. · Neural Machine Translation...

NLP Applications using Deep Learning

Giri Iyengar

Cornell University

gi43@cornell.edu

Feb 26, 2018

Giri Iyengar (Cornell Tech) NLP Applications Feb 26, 2018 1 / 27

Agenda for the day

Sentiment AnalysisNeural Machine TranslationEntailment

Overview

1 Sentiment Analysis

2 Neural Machine Translation

3 Textual Entailment

Sentiment Analysis

Defining Sentiment AnalysisSentiment Analysis is the process of determining whether a piece ofwriting is positive, negative or neutral. It is also known as opinion mining,deriving the opinion or attitude of a speaker.

Common Use case: How people feel about a particular {Brand, Topic,News, Company, Product}?

Sentiment Analysis: Naive Bayes

Bag of wordsUse words without caring for their order as features. For each word, useits vector representation (e.g. word2vec, GloVe)

Sentiment Analysis: Logistic Regression

Sentiment Analysis using Bag of Words

Context is very importantWord order conveys informationRedundancy in human language

Recurrent Neural Networks for Sentiment Analysis

Word order is naturally captured in this model

Entire sentence/phrase is consideredVariable lengths of inputs are naturally handled without artificialpadding / chopping

Word order is naturally captured in this modelEntire sentence/phrase is considered

Variable lengths of inputs are naturally handled without artificialpadding / chopping

Word order is naturally captured in this modelEntire sentence/phrase is consideredVariable lengths of inputs are naturally handled without artificialpadding / chopping

Sentiment Analysis: RNN

Figure: Source: A. Karpathy

Many to One RNNFeed words of a sentence in and then run a classifier on the finalrepresentation

Sentiment Analysis: RNN

Figure: Source: deeplearning.net

Average pooling over timeFeed words of a sentence in and then average pooling over time to classify

Sentiment Analysis: CNN

Figure: Source: Y. Kim, NYU

Sentiment Analysis using CNNStart with sentence matrix. Consider k words at a time, where k varies.Pool and classify.

Sentiment Analysis using CNNs and RNNs

Both CNN and RNN can use static word representations

Or, they could learn/fine-tune these representationsCan make use of multiple representations. Not just oneWinner has been switching places time and again

Both CNN and RNN can use static word representationsOr, they could learn/fine-tune these representations

Can make use of multiple representations. Not just oneWinner has been switching places time and again

Both CNN and RNN can use static word representationsOr, they could learn/fine-tune these representationsCan make use of multiple representations. Not just one

Winner has been switching places time and again

Both CNN and RNN can use static word representationsOr, they could learn/fine-tune these representationsCan make use of multiple representations. Not just oneWinner has been switching places time and again

Sentiment Analysis using Recursive Networks

Both CNNs and RNNs seem to not capture the compositionalstructure in language

Historically, sentences have been represented by ComputationalLinguists using Parse TreesThese trees seem to be a more natural representation of a sentencestructureCan we build models that explore/exploit this structure?

Both CNNs and RNNs seem to not capture the compositionalstructure in languageHistorically, sentences have been represented by ComputationalLinguists using Parse Trees

These trees seem to be a more natural representation of a sentencestructureCan we build models that explore/exploit this structure?

Both CNNs and RNNs seem to not capture the compositionalstructure in languageHistorically, sentences have been represented by ComputationalLinguists using Parse TreesThese trees seem to be a more natural representation of a sentencestructure

Can we build models that explore/exploit this structure?

Both CNNs and RNNs seem to not capture the compositionalstructure in languageHistorically, sentences have been represented by ComputationalLinguists using Parse TreesThese trees seem to be a more natural representation of a sentencestructureCan we build models that explore/exploit this structure?

Parse Trees

Figure: Parse Tree examples: Courtesy Wikipedia

Parse Trees

Recursive Networks for Sentiment Analysis

Figure: Source: Richard Socher/EMNLP2013

Overview

Neural Machine Translation using RNN

Figure: Image Source: http://cs224d.stanford.edu/lectures/CS224d-Lecture8.pdf

Feed entire sentence in source language

Extract sentence in target languageTrain using parallel corporaE.g. European Parliamentary proceedings. Canadian HansardEncoder-Decoder paradigm

Feed entire sentence in source languageExtract sentence in target language

Train using parallel corporaE.g. European Parliamentary proceedings. Canadian HansardEncoder-Decoder paradigm

Feed entire sentence in source languageExtract sentence in target languageTrain using parallel corpora

E.g. European Parliamentary proceedings. Canadian HansardEncoder-Decoder paradigm

Feed entire sentence in source languageExtract sentence in target languageTrain using parallel corporaE.g. European Parliamentary proceedings. Canadian Hansard

Encoder-Decoder paradigm

Feed entire sentence in source languageExtract sentence in target languageTrain using parallel corporaE.g. European Parliamentary proceedings. Canadian HansardEncoder-Decoder paradigm

Beam Search to generate target sentences

Figure: Image Source:https://deepage.net/img/beam-search/beam-search-opt.jpg

The entire sentence is squished into one vectorIncreasing the hidden state size is not a solution

Attention based NMT models

While translating, have access to the entire hidden stateEach word translation looks up the words emitted so far and parts ofthe source sentenceModel identifies parts of the source sentence that the decoder paysattention to while translating

Attention based NMT model

Figure: Image Source: Bahdanau, ICLR 2015

Overview

Textual Entailment

Textual Entailment is directional relation between text fragmentsEntailing and Entailed sentences are referred as text t and hypothesish respectivelyMeasures Natural Language Understanding because it requires asemantic interpretation of the text

Textual Entailment

An example of a positive TE (text entails hypothesis) is:text: If you help the needy, God will reward you.hypothesis: Giving money to a poor man has good consequences.

An example of a negative TE (text contradicts hypothesis) is:text: If you help the needy, God will reward you.hypothesis: Giving money to a poor man has no consequences.

An example of a non-TE (text does not entail nor contradict) is:text: If you help the needy, God will reward you.hypothesis: Giving money to a poor man will make you a better person.

NLP Applications using Deep Learning - GitHub Pages · 2018. 4. 29. · Neural Machine Translation...

Documents

Matching Natural Language Sentences with Hierarchical Sentence … · 2018-03-02 · Matching Natural Language Sentences with Hierarchical Sentence Factorization Bang Liu1, Ting Zhang1,

Recurrent Neural Networks (RNN)guerzhoy/321/lec/W08/rnn.pdf · RNN for Language Modelling •Given a list of word vectors (e.g., one-hot encodings of words) 1,…, 𝑡−1, 𝑡,

Zero-Shot Language Transfer for Cross-Lingual Sentence

Sentence Repetition Task as a Measure of Language Dominance

PSY 369: Psycholinguistics Language Comprehension: Sentence comprehension

Silliman Ron the New Sentence 1987 Poetry Language Poetry

RNN Explore

Simple Sentence Structure of Standard Arabic Language and Standard

AN RNN-BASED MUSIC LANGUAGE MODEL FOR … · AN RNN-BASED MUSIC LANGUAGE MODEL FOR IMPROVING AUTOMATIC MUSIC TRANSCRIPTION Siddharth Sigtia†, Emmanouil Benetos‡, Srikanth Cherla‡,

Machine Translation - nlp.cs.princeton.eduMT basics • Goal: Translate a sentence in a source language (input) to a sentence in the target language (output) • Can be formulated

DiSAN: Directional Self-Attention Network for RNN/CNN … · · 2017-11-22DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding ... e.g., word/character

RNN & LSTM

RNN Slides

GLENCOE LANGUAGE ARTS Sentence Diagraming

Sentence Intonation for Polish Language

Psycholinguistics and sentence processingling.umd.edu/~omaki/teaching/Ling240_Summ2007/slides_L18_Sentence...Psycholinguistics and sentence processing LING240: Language and Mind, Summer

Multidimensional RNN

[Michigan English Language Institute] English Sentence Structures

Candidate sentence selection for language learning exercises ......Candidate sentence selection for language learning exercises: from a comprehensive framework to an empirical evaluation

Literal and Figurative Language; Determiners of a Sentence