Upload
mark-cieliebak
View
1.102
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Sentiment analysis appears to be one of the easier tasks in the realm of text analytics: given a text like a tweet or product review, decide whether it contains positive or negative opinion. This task is almost trivial for humans, but it turns out to be a true challenge for automated systems. In fact, state-of-the-art sentiment analysis tools are wrong on approx. 4 out of 10 documents. Current sentiment analysis tools are rule-based, feature-based, or combinations of both. However, recent research uses deep learning on very large sets of documents. In this talk, we will explain the intrinsic difficulties of automated sentiment analysis; present existing solution approaches and their performance; describe an architecture for a deep learning system; and explore whether deep learning can improve sentiment analysis accuracy.
Citation preview
Can Deep Learning solve the
Sentiment Analysis Problem?
Mark CieliebakZurich University of Applied Sciences
Annual Meeting of SGAICO – Swiss Group for Artificial Intelligence and Cognitive Science18.11.2014
Outline
1. What is sentiment analysis?
2. How good are "classical" approaches?
3. Does deep learning solve the problem?
Mark Cieliebak 218.11.2014
About Me
18.11.2014 Mark Cieliebak 3
Mark CieliebakInstitute of Applied Information Technology (InIT)
ZHAW, Winterthur
Email: [email protected], Website: www.zhaw.ch/~ciel
Text
Analytics
Open
Data
Automated
Test
Generation
Research
Interests
Software
Engineering
What is Sentiment Analysis
"… WiFi Analytics is a free Android app that I find very
handy when it comes to troubleshooting and monitoring
a home network. "[1]
18.11.2014 Mark Cieliebak 4
Sample Application:
Social Media Monitoring
Text Analytics Components:
• Find relevant documents
• Hot topic Analysis
• Sentiment analysis
18.11.2014 Mark Cieliebak 5
[7]
Flavours of Sentiment Analysis
• Document Based
• Sentence Based
• Target-Specific
• Rating Prediction
18.11.2014 Mark Cieliebak 6
Classic Approaches to Sentiment Analysis
Rule-Based Corpus-Based
18.11.2014 Mark Cieliebak 7
Predicted
Label
[3][4]
Simple Sentiment Analysis
Idea: Count number of positive and negative words
"This camera is great[+1]." +1 (pos)
"I find it beautiful[+1] and good[+1]." +2 (pos)
"It looks terrible[-1]." -1 (neg)
"This car has a blue color." 0 (neu)
POSITIVE:
great
love
nice
...
NEUTRAL:
hello
see
I
…
NEGATIVE:
bad
hate
ugly
...
Use Sentiment-Dictionary:
18.11.2014 Mark Cieliebak 8
Sample Rules
18.11.2014 Mark Cieliebak 9
• Detect Booster Words: "The car is really very expensive[-1 -1 -2] ."
• New Category "Mixed": "This car has an appealing[+1] design and
comfortable[+1] seats, but it is expensive[-1]. "
• Negation: Invert only score of words occuring after the negation:
"The car is appealing[+3] and I do not[*-1] find it expensive[-2]"
• I do not find the car expensive and it is appealing.
Need to “understand” the sentence
Linguistic Analysis
-> RULE: Invert scores of words being in the same phrases as negation.
“I do not find the car expensive[+2]
and it is appealing[+3].” → +5 (pos)
Sentence
Sentence Conj. Sentence
Noun
Phrase
Verb Phrase
Verb Adverb Verb Noun
Phrase
Adj. Noun
Phrase
Verb Phrase
Det. Det Noun Det. Verb Participle
I do not find the car expensive and it is appealing
18.11.2014 Mark Cieliebak 10
Rule-Based Sentiment Analysis
Most Important Issues:
- Requires good hand-crafted rules
- Hard to transfer to new tasks
or languages
- Does not work well for texts with bad grammer (Twitter)
18.11.2014 Mark Cieliebak 11
[5]
Classic Approaches to Sentiment Analysis
Rule-Based Corpus-Based
18.11.2014 Mark Cieliebak 12
Predicted
Label
[3][4]
Corpus-Based Sentiment Analysis
18.11.2014 Mark Cieliebak 13
Predicted
Label
[4]
Corpus-Based Sentiment Analysis
Annotated Corpus
Sentence Polarity
This analysis is good. Pos
It looks awful. Neg
This car has a blue color. Neu
This car has an appealing design, comfortable seats, but it is expensive. Mix
This car has a very appealing design, comfortable seats, but it is really expensive. Mix
This analysis is not good. Neg
This car has an appealing design, comfortable seats and it is not expensive. Mix
This movie was like a horror event. Neg
This car is appealing and is not expensive. Mix
... ...
18.11.2014 Mark Cieliebak 14
Sample Features for Tweets
• Word ngrams: presence or absence of contiguous sequences of 1, 2, 3,
and 4 tokens; noncontiguous ngrams
• POS: the number of occurrences of each part-of-speech tag
• Sentiment Lexica: each word annotated with tonality score (-1..0..+1)
• Negation: the number of negated contexts
• Punctuation: the number of contiguous sequences of exclamation marks,
question marks, and both exclamation and question marks
• Emoticons: presence or absence, last token is a positive or negative
emoticon;
• Hashtags: the number of hashtags;
• Elongated words: the number of words with one character repeated (e.g.
‘soooo’)from: Mohammad et al., SemEval 2013
18.11.2014 Mark Cieliebak 15
Corpus-Based Sentiment Analysis
Most Important Issues:
- Requires large annotated corpora
- Depends on good features
18.11.2014 Mark Cieliebak 16
[6]
How good are Sentiment Analysis Tools?
Mark Cieliebak 1718.11.2014
Quick Poll
• Short texts: 1-2 sentences from
Twitter, news, reviews etc.
• Three-class classification: positive,
negative, other
• Accuracy =# 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑑𝑜𝑐𝑠
# 𝑑𝑜𝑐𝑠
Mark Cieliebak 21
Accuracy Votes
<50%
50-60%
60-70%
70-80%
80-90%
>90%
"How good are state-of-the-art
sentiment analysis tools?"
18.11.2014
Tool Accuracy
0,2
0,3
0,4
0,5
0,6
0,7
0,8
Accu
rac
y
Best Tool per Corpus
Worst Tool per Corpus
22
61%
40%
Avg.
Mark Cieliebak18.11.2014
[14]
Tool Accuracy
0,2
0,3
0,4
0,5
0,6
0,7
0,8
Accu
rac
y
Best Tool per Corpus
Worst Tool per Corpus
Overall Best Tool
23
61%
40%
59%
Avg.
Mark Cieliebak18.11.2014
Take-Home Lesson
Accuracy of best commercial tool on
arbitrary short texts is 59%
18.11.2014 Mark Cieliebak 24
Approaches to Sentiment Analysis
Rule-Based Corpus-Based
18.11.2014 Mark Cieliebak 25
Pr
edi
cte
d
La
bel
[9]
Deep Learning
[8]
Deep Learning on Text
It's all about Word Vectors!
18.11.2014 Mark Cieliebak 26
Word2Vec
• Huge set of text samples
(billions of words)
• Extract dictionary
• Word-Matrix: k-dimensional
vector for each word (k typically
50-500)
• Word vector initialized randomly
• Train word vectors to predict
next words, given a sequence of
words from sample text
18.11.2014 Mark Cieliebak 27
Major contributions by Bengio et al. 2003, Collobert&Weston 2008, Socher et al. 2011, Mikolov et al. 2013
[9]
The Magic of Word Vectors
18.11.2014 Mark Cieliebak 28
King - Man + Woman ≈ Queen
Live Demo on 100b words from Google News dataset: http://radimrehurek.com/2014/02/word2vec-tutorial/
[10]
Relations Learned by Word2Vec
18.11.2014 Mark Cieliebak 29
[11]
Using Word Vectors in NLP
18.11.2014 Mark Cieliebak 30
Collobert et al., 2011:
• SENNA: Generic NLP System based on word vectors
• No manual feature engineering
• Solves many NLP-Tasks as good as benchmark systems
[12]
Deep Learning and Sentiment
Maas et al., 2011
• Enrich word vectors with sentiment context
• Capture semantic of words (unsupervised) and sentiment (supervised) in
parallel, using multiple learning tasks
wonderful
amazing
terrible
awful
18.11.2014 Mark Cieliebak 31
Deep Learning and Sentiment
Socher et al. 2013:
• Word Vectors do not help for Sentiment Analysis
• Recursive Neural Tensor Networks
• Representing sentence structures as trees while adding sentiment
annotations at same time
• Restricted to single, well-structured sentences
•
18.11.2014 Mark Cieliebak 32
[13]
Deep Learning and Sentiment
Quoc and Mikolov, 2014:
• "Paragraph Vectors"
• Add context (sentence, paragraph, document) to word vectors during
training
• Improves many existing approaches
18.11.2014 Mark Cieliebak 33
[9]
Does Deep Learning solve the
Sentiment Analysis Problem?
18.11.2014 Mark Cieliebak 34
Conclusion: Deep Learning for Sentiment
• Small improvements, not revolution
• Very recent research, not yet "end of the story"
• SemEval 2015 will be benchmark
18.11.2014 Mark Cieliebak 35
Talk in Short!
1. Classic approaches are rule-based
or corpus-based
2. State-of-the-art tools classify 4
out of 10 docs wrong
3. Deep Learning does not need
hand-crafted features
4. Deep Learning improves
existing benchmarks
Mark Cieliebak 3618.11.2014
Thank You!
Mark CieliebakZurich University of Applied Sciences (ZHAW)
Winterthur, Switzerland
Email: [email protected], Website: www.zhaw.ch/~ciel
18.11.2014 Mark Cieliebak 37
[15]