Upload
others
View
14
Download
0
Embed Size (px)
Citation preview
A MACHINE LEARNING APPROACH TO
SENTIMENT ANALYSIS AND STANCE
DETECTION FOR POLITICAL TWEETS EXPLORING THE INFLUENCE OF IRONY ON THE PREDICTABILITY
OF SENTIMENT AND STANCE
Aantal woorden: 14658
Lot De Kimpe Studentennummer: 01404202
Promotor: Prof. Dr. Els Lefever
Masterproef voorgelegd voor het behalen van de graad Master in de Meertalige Communicatie
Academiejaar: 2017 - 2018
Page 3 of 76
Verklaring i.v.m. auteursrecht
De auteur en de promotor(en) geven de toelating deze studie als geheel voor consultatie
beschikbaar te stellen voor persoonlijk gebruik. Elk ander gebruik valt onder de beperkingen
van het auteursrecht, in het bijzonder met betrekking tot de verplichting de bron uitdrukkelijk
te vermelden bij het aanhalen van gegevens uit deze studie.
Het auteursrecht betreffende de gegevens vermeld in deze studie berust bij de promotor(en).
Het auteursrecht beperkt zich tot de wijze waarop de auteur de problematiek van het onderwerp
heeft benaderd en neergeschreven. De auteur respecteert daarbij het oorspronkelijke
auteursrecht van de individueel geciteerde studies en eventueel bijhorende documentatie, zoals
tabellen en figuren.
Page 4 of 76
Abstract
With the emergence of Web 2.0, easy-accessible microblogging platforms such as Facebook
and Twitter have allowed users to easily share their opinions online. Sentiment analysis and
stance detection, allow a business, organization or political party to gather all these viewpoints
and to find out which sentiment (positive, negative or neutral) a piece of text contains to
optimize their products and services. Despite the fast developments in this field of study,
challenges for the automatic prediction of sentiment and stance labels are still present (Pang et
al., 2008; Kumar & Sebastian, 2012; Mandya et al., 2016). In this research, it was explored how
well a machine learning system performs for sentiment analysis and stance detection on an
English Twitter corpus of 482 political tweets with #Brexit. The manually annotated labels were
compared to the predictions of a machine learning system, considering the possible impact of
irony on the performance of our system. The results show that the system performs fairly well
on sentiment analysis (accuracy of 0,55) and stance detection (accuracy of 0,61). It remains,
however, unclear to which extent irony affects the quality of the automatic predictions. Further
research could specifically focus on the comparison between irony detection and sentiment
analysis or stance detection. (204)
Page 5 of 76
Acknowledgements
First of all, I would like to express my gratitude towards my sister, Lies De Kimpe. As from the
very first letter of my bachelor’s paper, she provided me with professional feedback and
encouraging pep talks. And even now, up until the very last letter of my master’s thesis, she has
always been by my side for support. Without her eye for detail and willingness to answer every
little question, this paper would have not reached the quality it has today.
Secondly, many thanks go to my supervisor, Els Lefever, for her help, patience and support
during the last two years. She has always given me useful advice and motivating compliments,
which encouraged me to complete my thesis successfully. Furthermore, I also wish to thank her
for being such an approachable and kind mentor.
Thirdly, I would like to thank my friends Julie Carton, Lien De Wulf, Bo Van Eetvelde and the
entire group of KLJ people, who were my towers of strength in stressful moments. They were
wonderful in offering my daily dose of distraction in solitary times behind my desk. Special
thanks go to Hanne Christiaens, who was willing to share her recognizable experiences as a
master’s student at VTC with me.
And last but definitely not least, I wish to thank my parents from the bottom of my heart for
allowing me to pursue any possible dream and keeping their endless faith in me.
Page 6 of 76
Table of contents
List of tables and figures ......................................................................................................................... 8
1. INTRODUCTION ........................................................................................................................... 9
2. LITERATURE STUDY ................................................................................................................ 11
2.1 Sentiment analysis ................................................................................................................. 11
2.1.1 Terminology .................................................................................................................. 12
2.2 Approaches to SA .................................................................................................................. 13
2.2.1 Lexicon-based approach to SA ...................................................................................... 14
2.2.2 Supervised machine learning approach to SA ............................................................... 15
2.3 ABSA: Aspect Based Sentiment Analysis ............................................................................ 17
2.4 Sentiment analysis for political tweets .................................................................................. 17
2.4.1 Twitter ........................................................................................................................... 18
2.5 Stance detection ..................................................................................................................... 19
2.6 Irony detection ....................................................................................................................... 20
2.6.1 What is irony? ............................................................................................................... 20
2.6.2 Difficulties and challenges ............................................................................................ 21
3. RESEARCH DESIGN .................................................................................................................. 22
3.1 Research questions and hypotheses ....................................................................................... 22
3.2 Methodology ......................................................................................................................... 23
3.2.1 Data collection ............................................................................................................... 23
3.1.1 Annotation ..................................................................................................................... 24
3.1.2 Experimental approach .................................................................................................. 25
4. RESULTS ...................................................................................................................................... 26
4.1 Results manual annotation ..................................................................................................... 26
4.1.1 Sentiment and topics...................................................................................................... 26
4.1.2 Stance and irony ............................................................................................................ 28
4.2 Results machine learning system ........................................................................................... 30
4.2.1 Sentiment and topics...................................................................................................... 30
4.2.2 Stance and irony ............................................................................................................ 31
4.3 Analysis ................................................................................................................................. 33
4.3.1 Sentiment analysis: tenfold cross-validation scheme .................................................... 33
4.3.2 Sentiment analysis: general overview ........................................................................... 34
4.3.2.1 Impact of irony on the prediction of sentiment ....................................................... 34
Page 7 of 76
4.3.2.2 Error Analysis .......................................................................................................... 34
4.3.3 Stance detection: tenfold cross-validation scheme ........................................................ 39
4.3.4 Stance detection: general overview ............................................................................... 40
4.3.2.1 Impact of irony on the prediction of stance ............................................................. 41
4.3.2.2 Error analysis .......................................................................................................... 42
4.3.5 Comparison sentiment analysis and stance detection .................................................... 43
5. CONCLUSION ............................................................................................................................. 46
6. LIMITATIONS AND FURTHER RESEARCH ........................................................................... 49
APPENDIX 1 ........................................................................................................................................ 54
APPENDIX 2 ........................................................................................................................................ 57
Page 8 of 76
List of tables and figures Table 1 - Sentiment per topic (manual annotation) ............................................................................... 28
Table 2 - Manual annotation sentiment, stance and irony ..................................................................... 30
Table 3 - Presence of irony per topic ..................................................................................................... 30
Table 4 - Sentiment per topic (machine learning approach) ................................................................. 31
Table 5 - Machine learning annotation sentiment, stance and irony ..................................................... 33
Table 6 - Tenfold cross-validation scheme for sentiment analysis ....................................................... 34
Table 7 - Comparison manual and automatic sentiment analysis + irony presence .............................. 35
Table 8 - Precision, recall, F1-score and accuracy of sentiment labels ................................................. 35
Table 9 - Tenfold cross-validation scheme for stance detection ........................................................... 40
Table 10 - Comparison manual and automatic stance detection + irony presence ................................ 40
Table 11 - Precision, recall, F1-score and accuracy of stance labels .................................................... 41
Table 12 - Comparison results sentiment analysis and stance detection ............................................... 44
Table 13 - Precision, recall, F1-score and accuracy of sentiment and stance labels ............................. 44
Table 14 - Accordance of manually assigned sentiment labels with their stance labels ....................... 45
Table 15 - Accordance of automatically assigned sentiment labels with their stance labels ................ 45
Figure 1 - Manual annotation sentiment ................................................................................................ 26
Figure 2 - Topics in tweets .................................................................................................................... 27
Figure 3 - Manual annotation stance ..................................................................................................... 29
Figure 4 - Machine learning annotation sentiment ................................................................................ 31
Figure 5 - Machine learning annotation stance ..................................................................................... 32
Page 9 of 76
1. INTRODUCTION
With the rise of Web 2.0, microblogging websites have increasingly become a valuable
platform for people to express their opinion and sentiment on a certain topic. Blogs, forums and
social media platforms allow users to easily add blogposts, reviews, reactions and ratings to
share their point of view on the internet. Since online opinionated texts offer a wide range of
easy accessible data, businesses, services and political parties are not bound to conduct surveys
or carry out opinion polls (Lui, 2012) to gather the public’s sentiment anymore. Nowadays,
organisations have enough feedback at their disposal to examine how people’s views differ on
a certain product, service or policy.
The system used to extract and analyse the public’s opinion is called sentiment analysis (SA).
Opinion mining or sentiment analysis is defined as “the computational study of opinions,
sentiments and emotions expressed in text” (Kumar & Sebastian, 2012, p.2). A product or
service can be adapted based on the outcome of the sentiment analysis, regarding whether the
general opinion of a certain aspect is neutral, positive or negative. This is beneficial for the
quality of the product or service and therefore the consumer’s satisfaction. Sentiment analysis
can be applied for various purposes: to detect trends on the social media of a business, to
discover what the underlying reason for the success or failure of a certain product is, or to
predict the outcome of a referendum or elections. Whereas SA helps to determine the speaker’s
sentiment in a piece of text, stance detection (SD) aims at extracting the author’s opinion
towards a certain target or entity. SD thus encloses “the task of automatically determining from
text whether the author of the text is in favour or, against, or neutral towards a proposition or
target” (Mohammad & Kiritchenko et al., 2016, p.31). It can be used to reveal weak spots or
positive aspects of a target. A target may be a product, an aspect of a certain service, a person,
a political point of view, an organisation, a policy, et cetera.
Of all social platforms that offer a wide range of accessible opinions and sentiment, Twitter is
notably interesting for both researchers and marketers who can use tweets to easily collect the
opinion of a large audience. This social network is particularly instrumental for politicians and
political parties to evaluate their position within the political landscape. Tumasjan et al. (2010)
indicate that Twitter is used frequently to speculate about politics, since messages mentioning
a party tend to reflect the outcome of, for instance, an election. Consequently, they conclude
that political tweets plausibly give an indication of how the current offline political landscape
is divided.
Page 10 of 76
Besides the advantages of Twitter as an easy accessible source of user-generated data, the
platform has its own specific characteristics. For the very reason that the data is user-generated,
Twitter messages or tweets will often contain language phenomena that are typical for online
messaging (e.g. flooding, abbreviations, emoji’s). Furthermore, as users are limited to write a
message in under 280 characters they are obliged to phrase their messages creatively. As a
result, irony, sarcasm, metaphors or other figurative use of language frequently appear in
tweets, which is challenging to detect due to its creative character. Regardless of the challenging
nature of figurative language such as irony, Reyes et al. (2012) constructed an irony detection
model specifically for short online texts. This model provided valuable insights into figurative
language use on Twitter and tasks such as sentiment analysis. Moreover, Van Hee (2017)
explored the automatic detection of irony on social media and found that a machine learning
approach can lead to good performance in combination with a varied set of information sources.
In the future, this irony detection system could, however, be further optimised. In the framework
of SemEval-2015 Ghosh et al. (2015) explored the determination of sentiment in tweets
containing irony, sarcasm or metaphors. For this purpose, they measured the polarity of tweets
that use creative and figurative language. Their system appeared to be useful for further
research.
In this study, we want to make a contribution to the existing findings on automatic sentiment
analysis on Twitter. It will be further explored how well a machine learning system performs
for sentiment analysis and stance detection on a Twitter corpus of political tweets. Additionally,
we will attempt to provide more insight into the impact of ironic language in tweets on the
predictability of sentiment and stance labels. The manual annotation of sentiment in a Twitter
corpus containing 482 tweets with the hashtag Brexit (#Brexit) will be compared with the
predicted sentiment labels of a supervised machine learning system for sentiment analysis.
Apart from sentiment labels, the tweets will also be provided with a stance label to explore
whether a machine learning system is able to detect (implicit) stance. On the basis of the
collected and analysed data and experimental results, we will try to provide an answer to
following questions: Can we automatically predict sentiment with the help of sentiment
analysis? Which impact does irony have on the predictability of sentiment? Can a machine
learning system detect implicitly expressed stance? In further passages of this study, we will
give an in-depth overview of the current state-of-the-art of sentiment analysis, including ABSA,
stance detection and irony detection.
Page 11 of 76
2. LITERATURE STUDY
2.1 Sentiment analysis
People’s opinions on and reviews about a certain event, product or service can have a big
influence on future customers. The latter often turn to blogs, review platforms and social media
to gather information and advice to form their own opinions. Therefore, sentiment analysis (SA)
is an important field of study. By performing SA, it can be determined whether a piece of text
is neutral, positive or negative. To gain insight into for instance sales figures or election results
it is beneficial for companies and organisations to keep track of their online reputation.
Preferably, SA is performed by an automatic system, since manual annotation would appear to
be a time-consuming, intensive task.
The term sentiment analysis appeared for the first time in 2003 in a study by Nasukawa and Yi
(2003). In the same year, Dave et al. (2003) mentioned opinion mining in their work. Nowadays,
both terms are used interchangeably to denote the same field of study. More specifically, SA is
considered to be “the field of study that analyses people’s opinions, sentiments, evaluations,
appraisals, attitudes and emoticons towards entities such as products, services, organizations,
individuals, issues, events, topics, and their attributes” (Liu, 2012, p.7). As from the year 2003,
this field of study has been well-studied and is still constantly developing.
Nowadays, a range of applications are available to systematically organize opinions, reviews,
ideas on entities and events as well as products. Some applications, for example, arrange
reviews and ratings according to sentiment, and other recommendation systems are able to only
recommend products or advertisements with positive sentiment (Pang et al, 2008, p.8).
Additionally, SA may also be valuable for governments to detect possible opposed or negative
voices. Especially during times of elections, the latest technologies can assist in keeping an
overview of the various points of view on politicians, parties, bills, et cetera. Besides numerous
possibilities for commercial applications, several problems still rise. These need solutions and
therefore make SA a relevant field of study to explore thoroughly and in further detail.
In this study, an in-depth overview will be given of the SA-related terminology in section 2.1.1
and the two approaches to SA in chapter 2.2. In chapter 2.3, Aspect Based Sentiment Analysis
(ABSA) will be explained, whereas in chapter 2.4, the particular nature of SA for political
tweets will be discussed. Lastly, the aspects, difficulties and challenges of irony detection will
be discovered in chapter 2.5.
Page 12 of 76
2.1.1 Terminology
As SA has increasingly been discussed over the last decennia, it is important to keep a clear
overview of which terms are being used in the field of study of Natural Language Processing
(NLP). In literature the terms “opinion mining, review mining and appraisal attraction” (Kumar
& Sebastian, 2012, p.2) are used interchangeably with the concept of SA. According to Liu
(2012), the terms mentioned all differ from each other on a certain level. They can, however,
all be classified under the umbrella term of SA.
In the field of SA, it is essential to make a distinction between subjectivity and subjectivity
analysis on the one hand, and sentiment and sentiment analysis on the other hand. Pang et al.
(2008) believe subjectivity comprises everything that contains a personal opinion, such as
evaluation, emotions and speculations. Subjectivity analysis is used to distinguish facts
(objective sentences) from opinions and sentiment (subjective sentences). Furthermore,
subjectivity and sentiment do not have the exact same meaning. Sentiment implies the
expression of an opinion, which often reveals the speaker’s attitude towards a certain topic. Not
only subjective sentences contain sentiment. The following sentence “The battery of my
Bluetooth speaker only lasts for 8 hours”, is for example an objective sentence with negative
sentiment. The usage only reveals a sense of disappointment by the speaker. By means of
automatic SA, polarity, opinions, emotions and other subjective information can be
automatically detected and analysed (Desmet et al., 2014). It is often used to determine which
sentiment (positive, negative, neutral) belongs to a piece of text.
Moreover, two kinds of opinions can be distinguished, namely regular opinions and
comparative opinions (Liu, p.12). A regular opinion expresses sentiment on a specific feature
or aspect of a product, for example “The battery of my Bluetooth speaker lasts for a very long
time.”. A comparative opinion, however, compares different products or services based on
certain aspects they (don’t) have in common, for instance “My JBL speaker has a further
Bluetooth ranger than my old Philips speaker.”. Specific words that regularly express the same
kind of sentiment can be used to easily recognize opinions. These words are defined as
sentiment words or opinion words (Liu, 2012, p.12). Some opinion words are generally
positively used such as ‘beautiful, nice, lovely’ and others usually have a negative connotation,
such as ‘ugly, stupid, boring’. In addition, there are also sayings or proverbs who are
systematically classified under a certain polarity label. ‘To have a finger in every pie’ for
example, is generally used negatively.
Page 13 of 76
There are different ways to categorize sentiment. For one, SA often makes use of a binary
classification and only assigns the labels ‘positive’ and ‘negative’. This is also defined as
sentiment polarity classification (Pang et al, 2008, p17). In some cases, the degree of positivity
and negativity can be more thorough and complex. Therefore, Boertjes (2011) also defines a
non-binary classification system. When following this system, sentiment can then for example
vary on a scale going from ‘extremely dissatisfied’ to ‘extremely satisfied’. This is also called
multi-class classification (Kumar & Sebastian, 2012, p.4), which implies SA with more than
two sentiment categories. In the latter, the star system is also often applied, in which one star
expresses dissatisfaction and five stars satisfaction. Cambria et al. (2013, p.16) also distinguish
two common sentiment analysis tasks: polarity classification (supra) and agreement detection.
On the one hand, they add other examples to the binary polarity classification, such as ‘thumbs
up’ and ‘thumbs down’ or ‘like’ and ‘dislike’. On the other hand, they introduce agreement
detection as another example of a binary classification system. This task helps to determine
whether two texts hold the same opinion and should receive the same or opposite sentiment
labels.
2.2 Approaches to SA
Sentiment analysis is an interdisciplinary field of study which focusses on web mining
(extracting and analysing information on the web) as well as natural language processing (NLP
or computational linguistics). Kumar and Sebastian (2012) describe four levels at which
sentiment can be determined: feature level, word level, sentence level and document level.
Feature based SA on entity and aspect level (Liu, 2012, p.11) focusses on the opinion on
different aspects or features instead of the sentiment of entire paragraphs, sentences or phrases.
The sentence “My new smartphone has a great camera, but the sound quality is horrible.”, for
instance, cannot be labelled as entirely positive or negative because it reports on two features
of the same product. The evaluation of the sound quality of this new smartphone is negative,
but the camera is evaluated positively. Feature based SA is often seen as the most challenging
level of SA since two or more aspects can be covered within one sentence. On word level, the
polarity label is predicted for each separate word as each word is then considered as a separate
unit, which can hold different sentiment (Kolkur et al, 2015, p.2). According to Kumar and
Sebastian (2012), adjectives are the parts of speech that contain the most explicit sentiment. On
sentence level, a polarity label is given to each separate sentence. A neutral label usually equals
the fact that the sentence does not hold a personal opinion or that the sentence is both positive
Page 14 of 76
as negative which neutralizes the sentiment. For SA on document level, entire documents (e.g.,
product reviews) are labelled positively or negatively as a whole. This level of SA is not
applicable when the document covers multiple products or aspects.
Systems for automatic SA rely on one of two valid approaches for the prediction of a sentiment
label: either a lexicon-based approach or machine learning approach is applied.
2.2.1 Lexicon-based approach to SA
The lexicon-based approach to SA generally makes use of a “dictionary of opinion words to
identify and determine sentiment orientation (positive, negative or neutral)” (Zhang et al, 2011,
p.2). These opinion words are words that each have their own sentiment label, based on the
positive or negative connotation they commonly have. The dictionary, which is also called a
sentiment lexicon or opinion lexicon (Liu, 2012, p.12), is usually completed with synonyms
and antonyms of the opinion words.
However, Zhang et al. (2011) indicate that the lexicon-based approach would lead to a low
recall problem. Due to the specific nature of online language, it is impossible to add every
existing opinion word to the sentiment lexicon. In other words, some words will explicitly
express sentiment, but will not be picked up using the lexicon-based approach. For example, in
the following sentence ‘I haaaate going the dentist.’, the negative expression ‘haaaate’ will most
likely not be detected. These expressions change continuously and, therefore, adding them to
the opinion lexicon would appear to be an endless and time-consuming task.
Moreover, Kumar et al. (2012) explain that the lexicon-based approach does not work on
domain specific level. More specifically, this means that a certain word can have a positive or
negative label depending on the context or domain. The word ‘unpredictable’, for instance, can
be positive when it is used in a film review, but negative when it is used to evaluate the steering
behaviour of a brand new car. So far, lexicon-based methods are not yet able to interpret the
sentiment or meaning of a word depending on the situation. According to Khan et al. (2015), it
would be too labour intensive and time-consuming to manually label each opinion word per
specific context (e.g. film or car review).
Page 15 of 76
2.2.2 Supervised machine learning approach to SA
Machine learning allows computers to automatically carry out analysis tasks by means of self-
learning algorithms (such as Naïve Bayes and Support Vector Machines). As Zhang et al.
(2011) explain, these algorithms or sentiment classifiers are trained using features such as
unigrams or bigrams (sequences of one or two words, respectively). Machine learning systems
can then deduce rules or patterns based on manually annotated data to predict polarity labels of
unseen data. The model is thus built based on labelled pieces of texts (sentences, paragraphs,
documents), called training data.
During this training phase, the data is processed in the form of structured information or
“features” extracted from the text. On the basis of all this information, called “feature vectors”,
self-learning systems can then predict which combination of information results in which
polarity label. This approach is therefore referred to as “supervised learning, because the
classifier is given direction in terms of which are good or bad examples of the class.” (Taboada,
2016, p.6).
To set up a statistic machine learning system, data has to be pre-processed in advance on
different levels. Large pieces of text are split up in separate sentences (sentence splitting) and
these are then again broken down into words or tokens (tokenization). Another pre-processing
step can be Part-of-Speech tagging (PoS-tagging) which attributes the morphosyntactic
category to the corresponding word (adjectives, adverbs, nouns, verbs). To analyse various
word forms as a single item, lemmatisation can be used to group together inflected forms of a
word. The infinitives of verbs will then, for example, be recognized in their conjugated forms.
After the pre-processing phase, features (namely lexical features and syntactic features) can be
extracted from a certain text. This process is called feature extraction. Lexical features, on the
one hand, such as tokens or lemmas (E.g., the word ‘worse’ is derived from ‘bad’), can offer
insight in which words occur in a text. Syntactic features, on the other hand, give grammatical
information about the text, such as the syntactic categories of words (e.g. bad is an adjective).
2.2.3 Classification: Support Vector Machines
Shoeb and Ahmed (2017) state that data classification aims to classify data into categories in
the most efficient and productive way. The goal is then to predict the correct category for unseen
data. One regularly used supervised machine learning algorithm is a Support Vector Machine
(SVM) which can be applied for both classification and regression. According to Pang and Lee
Page 16 of 76
(2002), SVMs have been rather effective in comparison to other traditional text categorization
algorithms, showing better results than the Naïve Bayes method for example. By means of a
number of features, the classifier categorizes objects into one of two classes, represented by a
vector. In the context of Twitter, each feature could represent a single word found in a tweet.
Another application of SVMs could be to classify a set of documents into two sentiment groups:
positive or negative documents. The classification process would then be based on other
documents which have already received a positive or negative label. The goals of a SVM is to
train a system that classifies new unseen objects into a certain category. Apart from the
application of SVMs in machine learning approaches to sentiment analysis, an SVM is also
used for text classification tasks such as detecting spam. For this, the classification would rely
on a large corpus of e-mails or other documents which have already manually been marked as
spam or non-spam. Moreover, SVMs are used for the recognition of images, where the
algorithm attempts to recognize aspects or colours of an image.
More formally12, SVMs attempt to find a hyperplane that can divide a dataset into two classes.
A hyperplane is a line that linearly separates and classifies a set of data. The data points (or
feature vectors) that are the nearest to this hyperplane are called support vectors. These points
1 http://blog.aylien.com/support-vector-machines-for-dummies-a-simple 2 https://www.quantstart.com/articles/Support-Vector-Machines-A-Guide-for-Beginners
Page 17 of 76
are the hardest to classify. However, the further away data points are positioned from the
hyperplane, the more reliable the classification of these points are.
2.3 ABSA: Aspect Based Sentiment Analysis
Whereas regular SA determines whether a piece of text is positive, negative or neutral, ABSA
or Aspect Based Sentiment Analysis tries to trace back the target of an opinion. According to
Declercq et al. (2017), this comes down to a very fine-grained approach to SA. ABSA systems
attempt to detect all expressions of sentiment within a piece of text. On sentence level, this
means certain entities can be identified and then be paired up with the corresponding attribute.
In a review on the newest Iphone, the entity types ‘battery’ or ‘camera’ can for example be
linked to the right attribute label such as ‘price’ or ‘quality’ (Pontiki & Galanis et al., 2016,
p.20). The system then attempts to detect the main attributes (features) of the entity to make an
estimate of the average sentiment of a certain text per aspect. In other words, an overview is
given of the positivity or negativity of opinions for each single aspect mentioned (Pavlopoulos,
2014, p.2).
Pavlopoulus (2014) distinguishes three subtasks of Aspect Based Sentiment Analysis: aspect
term extraction, aspect term aggregation and aspect term polarity estimation. First of all, aspect
term extraction helps to detect words or phrases that indicate a certain aspect of the entity that
is being discussed (e.g. battery, camera). After this first step, the extracted words or phrases are
called aspect terms. Secondly, the next subtask is described as aspect term aggregation, which
means the system clusters aspect terms which are quite similar (such as ‘camera’ and ‘video
camera’). Lastly, during the aspect term polarity estimation, the system evaluates all aspect
terms and estimates the average sentiment of every aspect term or cluster of aspect terms.
2.4 Sentiment analysis for political tweets
With the help of SA, companies and services can gain insight in the perception and reception
of their product or service. It is a source of valuable customer feedback that can help companies
to fine-tune or make adaptions to their product by taking the results of the SA into account.
Moreover, social organizations might be interested to know people’s opinions on current
controversial or social debates. Intuitively, the domain where opinions differ regularly is
politics. As reported by Pak and Paroubek (2010), it may be profitable for political parties to
gain a perception of whether people support their party programme or not. Whether it involves
a new policy or upcoming elections, social media such as Facebook or Twitter are constantly
Page 18 of 76
booming with posts containing political views. Tumasjan et al. (2010) point out that,
specifically in the weeks leading up to a political event (such as elections), political themes are
clearly on many users’ minds. Even politicians themselves attempt to reach the electorate and
mobilize possible supporters by communicating via Twitter. In chapter 2.4.1 we will give an
overview of the particular characteristics of Twitter and why this could be an interesting social
platform to perform SA on.
2.4.1 Twitter
Social networks offer a wide range of accessible opinions and sentiment. Khan et al. (2015)
indicate that Twitter3 in particular has a great amount of data at their disposal. On this social
platform, users can share their opinions with the rest of the tweeting community in the form of
tweets (posts on Twitter) consisting of maximum 280 characters. Each day an estimated 60
million tweets are sent into the world. Twitter is therefore an easy-accessible and informative
platform for both researchers and marketers to collect sentiment of a large audience. According
to Liu (2002), tweets are easier to analyse thanks to their length, when compared to reviews for
example, because tweeters attempt to come to the point in a concise answer. Pak and Paroubek
(2010) also indicate Twitter users come from different social groups with varying interests and
backgrounds. Even though American users are prevailing, Kulshrestha et al. (2012) indicate
that the twitter audience is represented by users from all over the world (231 countries).
Therefore, it is possible to collect data and build a corpus in different languages.
Besides all the advantages of using Twitter data, the special characteristics of Twitter can cause
some specific problems. The language use is often informal and can contain typical
abbreviations or words which are only used in online messaging (eg., ‘lmao’, ‘lovvve’). In that
regard, an automatic SA system probably will not recognize any opinion words or positive
sentiment in a sentence such as “I loooove McDonald’s new hamburger!”, because ‘loooove’
is not picked up as the word ‘love’. Furthermore, tweets with the hashtag ‘not’ (#not) usually
hold an ironic or sarcastic message, as can be seen in the following sentence “I love it when my
train is delayed #not”. The hashtag makes the previous statement invalid, but when it is not
picked up by the SA system, the tweet will receive a positive label instead of the correct
negative label. From time to time, Twitter users tend to self-annotate their own usage of irony,
3 www.Twitter.com
Page 19 of 76
as is confirmed by Reyes et al. (2012). They then add the hashtag ‘irony’ (#irony) to point out
their ironic use of language.
In addition, emoticons or emoji’s that express a certain emotion can also give an entirely
different meaning to a tweet, such as in the sentence:
“Luckily, my train has arrived right on time as usual ☹”.
On first glance, the tweet seems to contain positive sentiment, however, the sad and unsatisfied
smiley indicates the statement is written in a negative tone. Moreover, retweets (the sharing of
someone’s tweet on one’s own profile), replies to other tweets (marked with an ‘@’-sign
followed by a username) and adding pictures or links can pose a challenge. Retweets and replies
can be misinterpreted when the tweets are read separately, because then there is a risk that
valuable context is lost.
Even though the phenomena mentioned above are challenging, there have already been several
attempts to tackle such problems. A study by Van Hee et al. (2014), for example, has shown
that feature extraction (as discussed in chapter 2.2.2) using a machine learning approach
performs better than SA systems using lexicon-based approaches. The results reveal that after
the extraction of specific Twitter features, the SA system performed very well (with an F1-score
of 86,28) on a Twitter corpus. After programming rules for flooding, for example, the word
‘loooove’ could be picked up in the previously mentioned sentence “I loooove McDonald’s
new hamburger!”.
2.5 Stance detection
Whereas SA aims at detecting the sentiment of an opinion in a piece of text, stance detection
(SD) is used to pick up whether someone is for or against the subject (target) being debated.
This target may be an organisation, product, service, person or policy. To decide whether an
author is for or against an issue, it is important to follow the reasoning rather closely. Mandya
et al. (2016) indicate the problem that posts containing rebuttal arguments are not clear enough
to be classified as ‘for’ or ‘against’ the main issue being debated. Posts are most often
independent or non-dialogic and thus all features for classification have to be derived from the
post itself. To facilitate stance classification, Mandya et al. (2016) state that topic-stance
features or topic terms can be automatically extracted. For the topic ‘gun control’ the terms
would include for instance ‘firearm’, ‘rifle’ or ‘license’. Each topic term is then associated with
the author’s stance towards that topic. In the sentence “Firearms are nothing but trouble.”, for
Page 20 of 76
example, ‘firearms’ would be associated with the topic ‘gun control’. Then, the stance towards
‘gun control’ would be negative.
The author’s stance or outlook towards the target is in favour if we can deduce from a piece of
text that he or she supports the target. Mohammad et al. (2016) observe different expressions
of favourability, such as simply supporting the target, repeating the positive stance of someone
else or opposing someone who is opposed to the target. Opposite expressions generally result
in negative stance and disapproval of the target. If there is no evidence found whether someone
is for or against a certain target, this does not necessarily mean the author is neutral. It can also
be the case that the stance from this piece of text simply could not be detected.
2.6 Irony detection
Barbieri and Saggion (2014) argue that computational creativity or the creative use of language
has been one of the most challenging topics of Artificial Intelligence (AI) and NLP nowadays.
Even though irony has received little attention in computational linguistics, it is considered to
be a vital and relevant aspect in fields of study such as SA. Therefore, irony detection has
become an increasingly discussed task. Irony detection is the task of automatically classifying
pieces of text into the classes ‘ironic’ or ‘non-ironic’. According to Reyes et al. (2012) the
automatic detection of irony could be relevant in various research areas, such as electronic
commerce, product tracking and online marketing. Van Hee (2017) completes the list with other
fields of study such as language psychology, sociolinguistics and cyberbullying detection.
For SA, the presence of irony can affect the outcome drastically. Classic sentiment analysis
tools are generally not sensitive to the use of irony. They will therefore perform less accurate
when applied to ironic utterances. To successfully detect irony, it is important to firstly define
the concept and possible subcategories of irony. Only then, the specific forms and
characteristics of irony that are susceptible to computational analysis can be identified. For that
reason, we will elaborate on what is understood as ironic in chapter 2.5.1. Then, secondly, the
difficulties and challenges of irony detection will be covered in chapter 2.5.2.
2.6.1 What is irony?
Irony is a creative use of language which is omnipresent in human interaction. Van Hee et al.
(2015) define irony as “an evaluative expression whose polarity (i.e., positive, negative) is
changed between the literal and the intended evaluation, resulting in an incongruence between
the literal evaluation and its content.”. Since the language use is figurative, ironic pieces of text
Page 21 of 76
should not be interpreted literally. A more complex approach is thus required to also detect the
correct context or make associations with common knowledge.
As irony is a playful way to express oneself it comes in many different varieties. Kreuz and
Roberts (1993) already made a distinction between three varieties of irony: Socratic or dramatic
irony, irony of fate and verbal irony. Firstly, Socratic irony or dramatic irony describes the
tension between the hearer’s knowledge and what the hearer pretends to know. Here, ignorance
is sometimes feigned in order to reveal the errors in someone’s viewpoint or argument. An
example of Socratic irony could be a situation in which a parent is aware that his/her child has
come home after curfew. Instead of confronting the child with the facts, the parent will ask a
series of seemingly innocent questions that will eventually result in a confession. Secondly,
irony of fate, is explained as an incongruence between two situations. It is also referred to as
situational irony as the situations that are being discussed fail to meet some expectations. An
example of this could be a no-dog sign in an animal shelter. Lastly, if someone uses verbal
irony, the speaker intentionally implies the opposite of what he or she believes. Reyes et al.
(2012) only make a distinction between the two broad categories of verbal irony and situation
irony. Karoui et al. (2017), however, retain eight different categories. The first category covers
analogies, metaphors and comparisons which aligns two things with contrasting or different
concepts or domains. Secondly, the category of hyperboles and exaggeration enlarges a
situation to lay emphasis on a point. Thirdly, Karoui et al. (2017) also distinguish euphemisms,
which are phrases or words that help to soften reality. The fourth category contains rhetorical
questions and the fifth context shifts. The latter covers an abrupt change of the topic or tone of
the conversation. False assertions, the sixth category, are declarations that conflict with
common sense. Then, whereas a false assertion is implicit, an oxymoron or paradox (the seventh
category) explicitly expresses the contradiction. In the last category, all other expressions
containing situational irony are covered.
Furthermore, other figurative uses of language such as sarcasm need to be distinguished. Even
though there is an overlap in usage, they differ in “usage, tone and obviousness” (Reyes et al.,
p.260, 2013). Sarcasm, for example, has a higher level of aggressiveness to it. According to
Van Hee (2017), it is often used with the intention to hurt a target directly and intentionally. In
comparison with irony, the intensity is greater due to the combination of ridicule and negativity.
Irony is considered to be more so subtle and therefore more sophisticated.
2.6.2 Difficulties and challenges
Page 22 of 76
Since irony is already such a complex concept on its own, Reyes et al. (2013) indicate that it
would be unrealistic to set hopes on a single computational silver bullet for irony. More specific,
the lack of facial expression and vocal intonation in ironic tweets makes it a challenging task to
automatically detect irony. Considering that irony touches on almost every aspect of language,
a multidimensional approach for detecting irony in Twitter is desirable.
Furthermore, as already mentioned earlier (cf. chapter 2.4.1), micro-bloggers in Twitter
regularly add the hashtag irony (#irony) to indicate their use of it. Some speakers are aware
their use of language is ironic, however, Wang (2013) found Twitter users make no distinction
between irony and sarcasm. Reyes et al. (2013) affirm that users who add #irony to their tweet
merely have a diffuse and vague idea of what it is understood as an ironic text. Furthermore,
Van Hee (2017) found that one in five tweets carrying the hashtag were not ironic. As a result,
it can be concluded that manual annotations are of help training automatic irony detection
systems.
Current machine learning approaches to irony detection, for example the system of Van Hee
(2017), show that ironic tweets that hold a polarity contrast are more likely to be identified than
other types of irony. Yet, what remains a challenge is the detection of implicit sentiment (E.g.,
situations that have a specific positive or negative connotation, such as ‘going to the dentist’ or
‘hearing your train is delayed’). Experiments in this research, however, reveal “that analysing
tweets about a concept or situation appears to be a viable method to determine implicit
sentiment related to that concept or situation.” (Van Hee, p. 124, 2017).
3. RESEARCH DESIGN
In this study, we attempt to explore how well a machine learning system performs for SA on
Twitter. The following chapters will offer insight into the data collection and annotation of the
corpus. We will then discuss and analyse the results in chapter 4.
3.1 Research questions and hypotheses
This research aims at answering the question of how well a machine learning approach to SA
performs on a Twitter corpus of political tweets. Additionally, we will attempt to provide more
insight into whether the presence of ironic language in tweets influences the predictability of
tweets. To formulate a well-founded answer to these questions, we will firstly try to answer
following sub questions: Can we automatically predict sentiment with the help of sentiment
Page 23 of 76
analysis? Which impact does irony have on the predictability of sentiment and stance? Can a
machine learning system detect implicitly expressed stance?
Based on previous studies (cf. chapter 3) we can formulate the following hypotheses: We expect
our machine learning system to deliver reliable results for SA on a political Twitter corpus.
Presumably, they will not yet be able to detect and interpret ironic language use correctly in
most cases.
3.2 Methodology
To answer our research question(s), a Twitter corpus of 482 tweets was built and the tweets
were manually annotated with the labels positive, negative or neutral. The same tweets were
then annotated by a machine learning system for SA. The results of both the manual labelling
and automatic labelling will be compared and analysed in the results section (cf. chapters 4.1,
4.2, 4.3). In this section, we will discuss in detail how our data were collected and annotated.
3.2.1 Data collection
As one of the first steps in the data collection process, we decided on which topic we would
collect tweets. We chose to gather English tweets with the hashtag Brexit (#Brexit) of the 24th
of June 2016. Since the Brexit referendum was held the day before, on the 23rd of June, we
believed the day after the outcome would provide us with divided opinions. The referendum
decided on whether the UK would leave the European Union or not. 51,9% of the voters
appeared to be in favour of leaving the EU and won, whereas an almost equally large group
(48,1%) was against the Brexit and lost. Ever since the announcement of the Brexit referendum,
it has been a highly discussed topic within the European political landscape.
We included every tweet of the 24th of June 2016 with #Brexit in chronological order, but
decided to ignore tweets that appeared twice or more. Furthermore, we decided to discard tweets
with double opinions. An example of such a tweet would be “Scary stuff! Still #brexit is best!!”.
In the first part of the tweet, ‘scary stuff’ could be interpreted as negative, however the second
sentence clearly contains positive sentiment. The same conclusion can be drawn from the
following tweet “Britons will enjoy their victory today. But tomorrow, the hangover will be
fierce #Brexit #UKReferendum”. The first sentence is positive, whereas the second part warns
Page 24 of 76
for probable negative consequences of the Brexit. Since these contradictory sentiments could
easily confuse the classifier of the machine learning system, we decided to exclude tweets with
double opinions from the corpus. Our aim was to collect circa 500 tweets and originally our
corpus consisted of 512 tweets with #Brexit. After removing tweets that appeared twice or
more, or tweets with double opinions, we eventually retained 482 tweets.
3.1.1 Annotation
For the manual annotation of the Twitter corpus we focussed on four categories: sentiment,
topic/aspect, stance and irony. The purpose of these four categories was to gain insight into the
nature of the tweet, considering various approaches and classifications. In the first category,
sentiment, we looked at the tweet as a whole and decided on whether the content was positive,
negative or neutral, regardless what the actual subject of the tweet was. If the opinion expressed
in the tweet was indefinable, we attached a neutral label to the piece of text.
Furthermore, we manually classified all tweets into various topics or aspects, the second
category. Eventually we narrowed 72 topics (cf. Appendix 1) down to eight categories of topics:
Brexit, celebrities/politicians, economy, EU, Scottish referendum, Trump, USA and other. We
decided to focus on a small number of topics that serve as an umbrella term for several aspects,
to keep an convenient overview of the themes discussed. Under the topic
‘celebrities/politicians’ for example, we classified all tweets that mentioned a specific name of
a public or political figure (e.g., Lindsey Lohan, Boris Johnson, Nigel Farrage). The topic
‘other’ served as an undefined category to classify numerous dissimilar subjects (e.g., personal
information, jokes).
The annotation of the third category concentrated on the stance expressed by the author towards
the Brexit. As already mentioned in chapter 2.4.2, SA aims at detecting the sentiment of an
opinion in a piece of text, whereas stance detection is used to pick up whether someone is for
or against the subject (in this case: the Brexit) being debated.
Lastly, to explore which impact irony has on the predictability of sentiment, we added a fourth
category in the annotation process. Based on the eight different categories of irony Karoui et
al. (2017) distinguished (analogies, metaphors and comparisons; hyperboles and exaggerations;
Page 25 of 76
euphemisms; rhetorical questions; context shifts; false assertions; oxymorons or paradoxes;
situational irony), we marked tweets with ironic messages.
3.1.2 Experimental approach
For the experimental component of this thesis, a machine learning model was created to label
both the sentiment and stance of each tweet automatically. In total, two experiments were thus
carried out: one to predict sentiment and one to predict stance. Firstly, the Twitter corpus was
split up in, on the one hand, the separate tweets and, on the other hand, their manually annotated
stance and sentiment label. The tweets were separated from the labels by tab.
Secondly, all tweets were tokenised with LeTs, a multilingual linguistic pre-processing toolkit
developed by Van de Kauter et al (2013). This toolkit includes Part-of-Speech taggers,
lemmatizers and named entity recognizers. For this study, all tokens (words, punctuation,
numbers, symbols) were separated from each other with the pre-processing tool.
Thirdly, several features were extracted: unigrams, bigrams, trigrams, character n-gram features
with a range of 3-4 tokens, the number of flooded tokens, the number of flooded punctuation
tokens, the number of capitalized tokens and a sentiment lexicon look-up. The lexicon used is
called AFINN4, which is a list of English words for valence with an integer between minus five
(negative) and plus five (positive). With this sentiment lexicon, the number of positive, negative
and neutral tokens as well as the overall value of one tweet can be extracted.
Lastly, the Support Vector Machine was run and tested on a tenfold cross-validation scheme,
which means that 90% of the Twitter corpus was used as a train fold and 10% as a test fold.
This process was repeated ten times: each time with another 10% as test corpus up until the
moment that the entire corpus has served as test fold. The machine learning experiments were
carried out with LIBSVM5, which is an integrated software for support vector classification,
regression and distribution estimation. For this study, the linear kernel was used.
4 http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010 5 https://www.csie.ntu.edu.tw/~cjlin/libsvm/
Page 26 of 76
17%
53%
30%
FIGURE 1 - MANUAL ANNOTATION SENTIMENT (N=482)
Positive (84) Negative (254) Neutral (144)
Figure 1 - Manual annotation sentiment
4. RESULTS
In this chapter, we will discuss both the results of the SA in the manually annotated Twitter
corpus (cf. chapter 4.1) and the outcome of the automatically annotated Twitter corpus (cf.
chapter 4.2). Then, the two result sections will be compared and thoroughly analysed (cf.
chapter 4.3).
4.1 Results manual annotation
4.1.1 Sentiment and topics
The Twitter corpus consisted of 482 tweets with the hashtag Brexit (#Brexit). The manual
annotation resulted in 84 tweets with a positive sentiment label (17%), 254 tweets with a
negative sentiment label (53%) and 144 (30%) with a neutral sentiment label.
As already mentioned (cf. chapter 3.2.2), we made a distinction between 8 categories of topics
(cf. Figure 2). The greater part of the tweets (57%) addressed the topic of the Brexit itself. This
could be explained by the specific hashtag in every tweet (#Brexit). Then, several users also
mentioned the consequences for the economy after the Brexit outcome (7%). To nearly the
same extent, users tweeted about celebrities or politicians, using the hashtag Brexit (6%). Only
a negligible number of tweets (2 out of 482 tweets) considered a Scottish referendum, however,
the USA (4%) and Trump (4%) were discussed more frequently. Both the USA and Trump
were equally discussed within the context of the Brexit, mostly as a political point of
Page 27 of 76
57%
6%
7%
3%0%4%
4%
19%
FIGURE 2 - TOPICS IN TWEETS (N= 482)
Brexit (273) Celebrities/politicians (29) Economy (34)
EU (13) Scottish referendum (2) Trump (21)
USA (21) Other (89)
Figure 2 - Topics in tweets
comparison. American elections were then to be held in November 2017, which was a highly
debated topic at that time as well. An example of this would be: “Britons voted to strengthen
their borders. Will you do the same in November? #Brexit”. Besides discussing the referendum
or the outcome of the referendum, many users gave personal information or any other remarks
in their tweets. This resulted in a fair number of tweets commenting upon other topics or
considering personal information (19%). Following tweet is an example of such a tweet: “A
very good cereal served at my amazing property in Turnberry! #Brexit of Champions, just like
me! Enjoy!”.
By comparing the previous results (cf. Figure 1 and Figure 2) and joining them together (cf.
Table 1), it is noticeable that, with the exception of the category ‘other’, negative labels are
predominant in nearly every topic. Especially when discussing the Brexit and its outcome itself,
the tweets are notably negative. In the category ‘other’, there are 34 negative and 37 neutral
tweets. An explanation for this could be the presence of personal information, which tends to
be either critical or merely narrative. An example of the latter could be the following tweet: “I
know it is not good for me, but, on days when Britain chooses to #Brexit, I like to drink a Coke
and eat a cookie. #EURefResults”. In this tweet, the actual subject is not the Brexit but the
author’s diet. The sentiment is negative even though the tweet considers ‘Coke and cookies’.
Besides personal information, jokes are also omnipresent in the category ‘other’, as can be
observed in the following tweet: “Is this going to affect my chances of getting into Hogwarts?
Page 28 of 76
#Brexit”. Here, the sentiment is neutral, because the author does not express itself negatively
towards the Brexit, but simply jokes around.
TOPIC NEGATIVE NEUTRAL POSITIVE
Brexit (273) 153 (56%) 74 (27%) 46 (17%)
Other (89) 34 (38%) 37 (42%) 18 (20%)
Economy (34) 17 (50%) 11 (32%) 6 (18%)
Celebrities (29) 19 (66%) 8 (28%) 1 (34%)
USA (21) 7 (33%) 7 (33%) 7 (33%)
Trump (21) 17 (81%) 1 (5%) 3 (14%)
EU (13) 7 (54%) 4 (31%) 2 (15%)
Scottish referendum
(2)
0 (0%) 2 (100%) 0 (0%)
TOTAL 254 144 84 Table 1 - Sentiment per topic (manual annotation)
4.1.2 Stance and irony
To explore whether the author is for or against the subject of the Brexit, we also labelled the
stance of a tweet as positive, negative or neutral. In Figure 3, an overview is given of the stance
expressed in our Twitter corpus. The majority of the authors (288 tweets) expresses itself
negatively towards the Brexit (60%), whereas 26% has a rather neutral stance (124 tweets).
Moreover, 14% takes a positive stance on the referendum (70 tweets).
In some tweets, the stance is rather implicit, for instance in the following tweet: “So, that was
the dress rehearsal. Now that you Leavers have seen the effects of your vote, would you like to
try that again? #Brexit”. Even though the author does not explicitly express itself for or against
the Brexit, it is clear that he/she mocks with the leave-voters and is against Britain leaving the
EU. Tweets can also contain implicit positive stance, such as in the following tweet: “Leftists
are freaking out over the #Brexit. Why, because the people finally rejected tyranny? Just proves:
liberals = tyrants. #tcot”. The author is fairly negative towards the people who voted to remain
part of the EU and is therefore a supporter of the Brexit, which results in a positive stance label.
Page 29 of 76
Figure 3 - Manual annotation stance
If we compare the results of the SA with those of our stance detection (cf. Table 2), we find that
218 tweets (out of 482) hold negative sentiment as well as negative stance. Additionally, 43
tweets contain positive sentiment and stance, whereas 89 tweets have neutral sentiment and
stance. In total, 350 tweets (73%) have the same sentiment and stance label, which means 132
tweets show differences in the annotation of sentiment and stance. An example of the latter
could be the following tweet: “The only good thing to come out of the #Brexit is the dearth of
insults being hurled at @realDonaldTrump by the lovely people of Scotland”. Here the
sentiment of the tweet is positive, because the author explains a positive result of the Brexit.
The stance expressed, however, is negative, for the reason that the author does not see any other
positive consequences of the Brexit apart from new insults about Trump.
It is noticeable, that besides the relatively frequent use of irony in tweets (9%) with negative
sentiment and stance, especially tweets with neutral sentiment holding either negative or neutral
stance also contain ironic language (4%). In chapter 4.3, we will further explore whether a
machine learning system for sentiment analysis and stance detection will be influenced by the
usage of irony. As can be drawn from the overview in Table 3, irony is particularly used in
tweets specifically considering the economic consequences of the Brexit, followed by tweets
about Trump and the Brexit itself.
14%
60%
26%
FIGURE 3 - MANUAL ANNOTATION STANCE (N=482)
Positive (70) Negative (288) Neutral (124)
Page 30 of 76
SENTIMENT STANCE TOTAL
TWEETS PERCENTAGE
IRONIC
TWEETS
IRONY
PERCENTAGE
NEGATIVE NEGATIVE 218 45% 47 22%
NEUTRAL 19 4% 3 16%
POSITIVE 10 2% 0 0%
NEUTRAL NEGATIVE 45 9% 19 42%
NEUTRAL 89 18% 19 21%
POSITIVE 10 2% 0 0%
POSITIVE NEGATIVE 25 5% 8 32%
NEUTRAL 16 3% 4 25%
POSITIVE 43 9% 2 5%
= 482 = 103 Table 2 - Manual annotation sentiment, stance and irony
4.2 Results machine learning system
The second step in the experimental part of this study
consisted of the automatic annotation of the 482 political tweets with the hashtag Brexit. In this
section, we will discuss the results of both the sentiment and stance labels. In chapter 4.3, we
will compare the results of the manual with those of the machine learning approach.
4.2.1 Sentiment and topics
As can be drawn from the pie chart below (cf. Figure 5), the greater part (69%) of the Twitter
corpus consists, according to the SA-tool, of negative tweets. The following tweet, for example,
was picked up by the system and labelled as negative: “Still so sad about #Brexit. What is this
dark, absurd future being carved out for the world?”. Here, the sentiment words ‘sad’ and ‘dark’
were presumably a deciding factor. 117 tweets or 24% of the corpus received a neutral label
and a small 7% of the tweets was labelled as positive. An example of the latter could be:
“Learning some great new swears, thanks Scotland! #Brexit”. In this tweet, the author is
delighted to learn new insulting language phenomena from Scotland.
TOPIC NUMBER OF TWEETS
CONTAINING IRONY
IRONY PERCENTAGE
PER TOPIC
Brexit (274) 60 22%
Celebrities/politicians (29) 6 21%
Scottish referendum (2) 0 0%
Economy (34) 9 26%
EU (14) 2 14%
other (94) 19 20%
Trump (21) 5 24%
USA (22) 2 9%
= 103 Table 3 - Presence of irony per topic
Page 31 of 76
7%
69%
24%
FIGURE 4 - MACHINE LEARNING ANNOTATION
SENTIMENT (N=482)
Positive (34) Negative (331) Neutral (117)
Figure 4 - Machine learning annotation sentiment
As can be drawn from Table 4, the distribution of sentiment in each topic is not equally divided:
in every topic category, the negative labels are predominant, followed by considerably less
neutral and even fewer positive tweets. This can also be concluded from Table 1. Only in the
category ‘USA’, there is one more positive tweet than the neutral ones and in the category
‘Scottish referendum’, there are merely 2 tweets, which are both neutral. The majority of the
Twitter corpus consists of negative tweets on the Brexit referendum itself. An example of this
could be: “Never underestimate the power of stupid people in large groups! #Brexit
#jokeofthecentury”. In this tweet, the author utters itself negatively towards the Leave-voters
of the Brexit. The topic on ‘voters’ was classified under ‘Brexit’, as can be seen in Appendix
1.
TOPIC NEGATIVE NEUTRAL POSITIVE
Brexit (273) 190 (70%) 64 (23%) 19 (7%)
Other (89) 53 (60%) 30 (34%) 6 (6%)
Economy (34) 23 (68%) 9 (26%) 2 (6%)
Celebrities (29) 23 (80%) 4 (14%) 2 (7%)
USA (21) 14 (67%) 3 (14%) 4 (19%)
Trump (21) 17 (81%) 3 (14%) 1 (5%)
EU (13) 11 (85%) 2 (15%) 0 (0%)
Scottish referendum
(2)
0 (0%) 2 (100%) 0 (0%)
TOTAL 331 117 34 Table 4 - Sentiment per topic (machine learning approach)
4.2.2 Stance and irony
Considering the results of the stance detection returned by our classifier, it can be said that the
distribution of negative, neutral and positive stance is fairly similar compared to the distribution
Page 32 of 76
of sentiment (cf. Figure 5). However, the percentage of tweets with negative stance (79%) is
considerably higher than tweets with negative sentiment (69%). In Table 5, we can see that,
according to the automatic system for SA, 302 tweets contain both negative sentiment and
stance, which counts for 63% of all tweets. Of all these tweets, 13% percent contains ironic
language. With regard to the neutral labels, it can be drawn from Table 5 that 10% of the tweets
contain a neutral sentiment label as well as a neutral stance label. In 12% of all cases, tweets
with neutral sentiment received a negative stance label. Furthermore, it is remarkable that in
total there are very little entirely positive tweets (2%).
Figure 5 - Machine learning annotation stance
Concerning the percentage of ironic tweets, it can be stated that 32% of the tweets that received
a positive sentiment label and a negative stance label from the machine learning system contain
ironic language. Moreover, it is notable that a quarter of the tweets that contain positive
sentiment and negative stance are ironic. In absolute numbers, the category with both negative
sentiment and negative stance holds the highest number of ironic tweets, namely 64. In section
4.3.2.2, we will compare the accordance of sentiment with stance to the influence of irony in
further detail.
5%
79%
16%
FIGURE 5 - MACHINE LEARNING ANNOTATION
STANCE (N=482)
Positive (22) Negative (381) Neutral (79)
Page 33 of 76
SENTIMENT STANCE TOTAL
TWEETS
PERCENTAGE IRONIC
TWEETS
IRONY
PERCENTAGE
NEGATIVE NEGATIVE 302 63% 64 21%
NEUTRAL 22 5% 4 18%
POSITIVE 7 1% 1 14%
NEUTRAL NEGATIVE 60 12% 15 25%
NEUTRAL 50 10% 9 18%
POSITIVE 7 1% 2 29%
POSITIVE NEGATIVE 19 4% 6 32%
NEUTRAL 7 1% 1 14%
POSITIVE 8 2% 1 13%
= 482 = 103 Table 5 - Machine learning annotation sentiment, stance and irony
4.3 Analysis
In this chapter, we will compare and interpret the results of both the manual annotation and the
labelling of the machine learning system. We will verify how accurate the machine learning
system performs for sentiment analysis and stance detection on a Twitter corpus with political
tweets. Furthermore, we will analyse whether or not the presence of ironic language affects the
predictability of sentiment or stance.
4.3.1 Sentiment analysis: tenfold cross-validation scheme
In the methodology section (cf. 3.2) of this study, it was already explained that our machine
learning system was run and tested on a tenfold cross-validation scheme6. This means that 90%
of the Twitter corpus was used as a train fold and 10% as a test fold. This process was repeated
10 times to check the quality of the system. The tenfold cross-validation approach is found to
be reliable, for the reason that the entire corpus is used for both training and validation.
Moreover, each set is used for validation exactly once. In Table 10, an overview is given of the
results of our tenfold cross-validation scheme. In chapter 4.3.2, we will discuss the global
results of dataset as a whole.
It is noticeable that the scores can vary from fold to fold, considering that the cross-validation
process uses different train and test data in every fold. In Table 8, the ten accuracy scores are
similar to one another, whereas precision and recall scores differ notably. For example, the
precision scores of the positive labels in the various folds vary from zero to 0.67. The same
goes for the positive recall score: in Fold 01, for instance, the score is zero, whereas a score of
6 https://www.openml.org/a/estimation-procedures/1
Page 34 of 76
0.43 is given Fold 08. The F1 scores of the negative labels are all closely situated next to one
another, with only one peak in Fold 06 of 0.8. All negative F1 scores are well above 0.6, which
makes negative sentiment the best predicted label. The precision and recall of neutral labels
tend to lie far apart from each other in the different folds. In Fold 02, for instance, the neutral
precision score is 0,27, when in Fold 03 the score is remarkably higher (0.65).
Sentiment pos
prec
neg
prec
neutr
prec
pos
recall
neg
recall
neutr
recall
pos
F1
neg
F1
neutr
F1
accuracy
Fold 00 0.20 0.61 0.27 0.11 0.73 0.25 0.14 0.67 0.26 0.49
Fold 01 0 0.58 0.70 0 0.84 0.50 0 0.69 0.58 0.58
Fold 02 0.50 0.60 0.27 0.14 0.75 0.23 0.22 0.67 0.25 0.52
Fold 03 0 0.54 0.65 0 0.78 0.52 0 0.64 0.58 0.54
Fold 04 0.67 0.56 0.38 0.18 0.71 0.42 0.29 0.63 0.40 0.52
Fold 05 0 0.58 0.50 0 0.73 0.33 0 0.64 0.40 0.50
Fold 06 0 0.75 0.40 0 0.86 0.67 0 0.80 0.50 0.63
Fold 07 0 0.62 0.33 0 0.78 0.25 0 0.69 0.29 0.52
Fold 08 0.43 0.70 0.55 0.43 0.78 0.43 0.43 0.74 0.48 0.63
Fold 09 0.75 0.50 0.57 0.30 0.87 0.22 0.43 0.63 0.32 0.53
Table 6 - Tenfold cross-validation scheme for sentiment analysis
4.3.2 Sentiment analysis: general overview
In the first column of Table 7, an overview is given of the number of the manually given labels
per sentiment. The second column shows in yellow how many correct labels the machine
learning system assigned to the tweets. Besides the number of these true positives, an overview
is given of all false positives per sentiment category. The third column presents the number of
ironic tweets per automatically predicted label to provide insight into the effect of irony on the
predictability of sentiment
Page 35 of 76
MANUALLY
ANNOTATED
SENTIMENT
TWEETS THAT ARE
AUTOMATICALLY
PREDICTED AS
NUMBER OF
IRONIC TWEETS
(103)
IRONY
PERCENTAGE
NEGATIVE 254 NEGATIVE 199 (78%) 39 20%
NEUTRAL 40 (16%) 8 20%
POSITIVE 15 (6%) 4 27%
NEUTRAL 144 NEGATIVE 81 (56%) 21 26%
NEUTRAL 54 (38%) 16 30%
POSITIVE 9 (6%) 1 11%
POSITIVE 84 NEGATIVE 51 (61%) 9 18%
NEUTRAL 23 (27%) 2 9%
POSITIVE 10 (12%) 3 30% Table 7 - Comparison manual and automatic sentiment analysis + irony presence
To interpret the results presented in Table 7, we calculated precision, recall, and F1-score as
well as the accuracy of our machine learning system. Firstly, precision is the number of true
positives divided by the total number of true positives and false positives returned by the
classifier and is used to find out how “correct” the predictions per label are. Secondly, the recall
score reveals how many sentiment labels are predicted by dividing true positives by the total
number of true positives and false negatives returned by the machine learning system (per
label). Thirdly, the F1-score is the average of both precision and recall and shows how well the
system performs per sentiment category. The F1 or F-score is the result of the following
division: 𝑓 =2 . (precision.recall)
precision+recall . Lastly, the accuracy of the test set was calculated as a whole,
by dividing the number of correctly predicted labels by the total number of instances.
SENTIMENT Negative Neutral Positive
Precision 𝑝 =199
(199 + 81 + 51)
𝑝 = 0.6
𝑝 =54
(40 + 54 + 23)
𝑝 = 0.46
𝑝 =10
(15 + 9 + 10)
𝑝 = 0.29
Recall
𝑟 =199
254
𝑟 = 0.78
𝑟 =54
144
𝑟 = 0.38
𝑟 =10
84
𝑟 = 0.12
F1-score
f = 0.69 f = 0.42 f = 0.21
Accuracy 𝑎 =199 + 54 + 10
482
𝑎 = 0.55
Table 8 - Precision, recall, F1-score and accuracy of sentiment labels
Page 36 of 76
As can be drawn from Table 8, the accuracy of the machine learning system results in a score
of 0.55. This means that the performance of our SA system can be evaluated as moderate,
because the score is slightly above 0.5. The high precision score of the negative sentiment
shows that the system returned more true positives than false positives. Moreover, the high
recall score indicates that the SA system returned most of the relevant results. Yet, there is still
room for improvement, especially when looking at the precision and recall scores of the neutral
and positive sentiment labels. For example, the recall score of the positive labels shows that the
machine learning failed to return many of the positive tweets.
If we compare Table 8 with Table 6, we notice that our machine learning system for SA is rather
biased towards negative tweets. In other words, our classifier tends to overgeneralize negative
labels and attributes them falsely in 34% of the cases. Neutral labels are assigned falsely for
13% of the tweets and positive labels are only ascribed wrongly for 5% of the cases. This might
explain the high precision, recall and F1 scores within the negative category and the low results
for both positive and neutral labels. As a result of this difference between results for the negative
sentiment category on the one hand, and those for the positive and neutral sentiment category
on the other hand, the accuracy scores are only slightly above average.
From the third column in Table 7, it can be concluded that 45 of all 215 false positive cases
contained ironic language. In other words, irony was used in 21 percent of all falsely labelled
tweets. This could be an explanation for many of the errors, meaning that irony influences the
predictability of sentiment in political tweets. In chapter 4.3.1.3 we will discuss the various
errors in greater detail.
4.3.2.1 Impact of irony on the prediction of sentiment
In Table 7, an overview is given of how many tweets contain irony to analyse the possible effect
of creative language on the predictability of sentiment. In the fourth column, the irony
percentage in each sentiment category is presented. It can be noted, that in the negative
manually annotated category the irony percentages all lie between 20 and 27 percent. As a
result, it is difficult to conclude, whether the false positives (negative tweets that received a
neutral or positive label) are influenced by the presence of the ironic language or not, without
looking at the content of the tweets itself. For instance, in 20% of all negative tweets that
received a positive label, 27% contained irony. The same goes for the neutral and positive
sentiment category, where the true positives itself (e.g. neutral tweets that received a neutral
Page 37 of 76
label by our machine learning system) appear to contain the highest percentages of ironic
tweets. In total, if the amount of ironic false positives is divided by the total number of false
positives, it appears that 21% of the falsely labelled tweets by the machine learning system
contained irony. If we follow the same procedure for all true positives, it shows that 22% of the
correctly automatically labelled tweets also contained irony.
Considering the content of all falsely labelled tweets, it can, however, be concluded that irony
does have a certain effect on the machine learning system. In the following negative tweets, for
instance, the classifier ascribed a positive or neutral label due to irony:
(1) “Thanks British #brexit twats, I'm feeling poorer today.”
(2) “The next James Bond will just be him spending 2 hours in passport control De
Gaulle #Brexit #JamesBond”
(3) “Sebastian was right, can I become a mermaid now pls #Brexit”
(4) "They took back their country and that’s a great thing," Trump said of #Brexit, while
in Scotland IN SCOTLAND!!!!!
In tweet (1), our machine learning for SA does not detect the ironic expression of gratitude and
the negative connotation attributing ‘feeling poor’. In other words, the implicit sentiment in
‘feeling poor’ was not recognized. Moreover, the negative and insulting word ‘twats’ was not
picked up. Then, in tweet (2) the classifier decided on a neutral label, regardless of the fact that
spending 2 hours in passport control is a rather unpleasant activity. In the third tweet (3), a
hyperbole is used to express the author’s disbelief, with regard to the leave-voters in the Brexit
referendum. Here, some specific background knowledge is needed to understand the reference
made to the Little Mermaid. There, the character Sebastian attempts to warn mermaid Ariel for
the foolishness of human beings, by saying ‘The human world is a mess’. In (3), the author
confirms the truth in Sebastian’s reasoning. The system, however, fails to interpret the context
and irony in the tweet and assigns a neutral label. Lastly, tweet (4) is an example of situational
irony. The fact that the American president Trump expresses himself in favour of Britain ‘taking
their country back’ while being in Scotland is ironic, in the sense that Scotland itself has been
struggling in their search for independency for years. Here again, the system lacks the right
Page 38 of 76
contextual information to interpret the tweet correctly. Furthermore, the repetition of ‘in
Scotland’ in capital letters, followed by a punctuation flooding was not detected.
In chapter 4.3.4.1 we will go further into the possible effect of irony on our stance classifier.
4.3.2.2 Error analysis
Besides the effect of irony possibly causing the machine learning system to make errors, we
will consider another factor that may influence the automatic attribution of a wrong sentiment
label in this section. In some tweets, the machine learning system appears to have a notable lack
of common of specific knowledge. Therefore, following tweets were misinterpreted and
therefore falsely labelled:
(5) #Brexit, Monty Python & Silly Walks.
(6) Wales should have been more careful what it wished for. It's going to be given it.
#Brexit
(7) Proud to be #Brexit! Proud to stand alone in a world where most are too scared to
be alone, to have their own opinion. Proud to me! #UKref
In tweet (5), the classifier attributed a negative label, whereas it was originally granted a neutral
label. Presumably, the word ‘silly’ was picked up as a negative word. The tweet, however,
refers to the popular Monty Python sketch ‘Ministry of Silly Walks’, firstly broadcast in 1970,
which does not hold a negative connotation. The sixth tweet (6), contains an allusion to the
common expression ‘be careful what you wish for’. According to the Merriam-Webster7
dictionary, it is usually used ‘to tell people to think before they say that they want something
and to suggest that they may not actually want it’. Nonetheless, this negative tweet received a
neutral label. The last example (7), being alone is valued as a positive situation. The author
explains that he/she sees being alone as having an individual opinion, which is something to be
proud of. Therefore, the tweet is positive, but the classifier hands out a negative label because
it sees being alone as something negative.
7 https://www.merriam-webster.com/
Page 39 of 76
4.3.3 Stance detection: tenfold cross-validation scheme
To gain more detailed information on the accuracy of the stance labels returned by the classifier,
the tenfold cross-validation approach was once again applied on the Twitter corpus. In
comparison with the accuracy scores of the SA, it can be concluded that, in general, the
classifier returned better accuracy results for stance detection than for SA. This will be further
discussed in chapter 4.3.4. Furthermore, it is remarkable that in, for instance, Fold 05 the
accuracy is relatively low (0.45) when compared to the high score of 0.73 in Fold 09. Mainly,
these differences can be declared by focussing on the positive precision and recall scores.
Remarkably, the positive precision scores vary between zero and one. An explanation for this
phenomenon could be the fact that in a tenfold cross-validation scheme, only 10 percent of all
data is used as test data in each fold. Considering that only 10 positive stance labels were
predicted correctly, it is comprehensible that in some folds none of the corresponding tweets
appeared. This then results in a score of zero for both precision and recall.
In general, the positive recall scores are all fairly low, varying between zero and 0.22. In
comparison, however, the precision scores are higher than the recall scores in the positive stance
category. The higher precision score indicates a low false positive rate, but it can be concluded
from the low recall that, in general, very few results were predicted. As regards the neutral
accuracy, the precision scores are again rather varied. They run from a very low 0.22 in Fold
02 to a fairly high score of 0.67 in Fold 03. The neutral recall scores do not reach above average,
with the exception of Fold 08. Moreover, the negative recall scores are all fairly continuous and
all lie between 0.75 and an almost perfect score of 0.93. It is noteworthy that the negative
precision scores are, for example in Fold 03, rather average, whereas Fold 02 shows a peak of
0.76.
Page 40 of 76
Stance pos
prec
neg
prec
neutr
prec
pos
recall
neg
recall
neutr
recall
pos
F1
neg
F1
neutr
F1
accuracy
Fold 00 0.50 0.64 0.42 0.13 0.75 0.45 0.20 0.69 0.43 0.57
Fold 01 1 0.59 0.63 0.13 0.92 0.33 0.22 0.72 0.43 0.60
Fold 02 0.50 0.76 0.22 0.14 0.82 0.29 0.22 0.79 0.25 0.65
Fold 03 0 0.47 0.67 0 0.9 0.29 0 0.62 0.40 0.48
Fold 04 1 0.67 0.38 0.14 0.84 0.30 0.25 0.74 0.33 0.63
Fold 05 0.25 0.45 0.67 0.14 0.81 0.20 0.18 0.58 0.31 0.46
Fold 06 0 0.7 0.43 0 0.88 0.27 0 0.78 0.33 0.65
Fold 07 0.50 0.68 0.25 0.17 0.81 0.2 0.25 0.74 0.22 0.60
Fold 08 1 0.70 0.67 0.22 0.93 0.55 0.37 0.80 0.60 0.71
Fold 09 0.50 0.77 0.33 0.40 0.90 0.13 0.44 0.83 0.18 0.73
Table 9 - Tenfold cross-validation scheme for stance detection
4.3.4 Stance detection: general overview
As already explained in chapter 4.3.2, an overview (cf. Table 7) was made of both the manually
distributed labels, with in the second column the corresponding labels extracted by the machine
learning system. In the third column, the number of ironic tweets was listed per label to analyse
whether ironic language influences the predictability of stance or not. Furthermore, precision,
recall, F1 score and accuracy were once again calculated to provide more insight into the quality
of the stance labels returned by the system. Table 10 and 11 show results of the entire data set,
after 10-fold cross validation.
MANUALLY
ANNOTATED
STANCE
TWEETS THAT ARE
AUTOMATICALLY
PREDICTED AS
NUMBER OF
IRONIC TWEETS
(103)
IRONY
PERCENTAGE
NEGATIVE 288 NEGATIVE 246 (85%) 63 26%
NEUTRAL 33 (12%) 8 24%
POSITIVE 9 (3%) 3 33%
NEUTRAL 124 NEGATIVE 84 (68%) 19 23%
NEUTRAL 37 (30%) 6 16%
POSITIVE 3 (2%) 1 33%
POSITIVE 70 NEGATIVE 51 (73%) 3 6%
NEUTRAL 9 (13%) 0 0%
POSITIVE 10 (14%) 0 0% Table 10 - Comparison manual and automatic stance detection + irony presence
Page 41 of 76
As can be concluded from Table 10 and 11, the most accurate predictions were made for the
negative stance category. Both precision and recall show reasonably good results with
correspondingly a score of 0.65 and 0.85, which results in an F1-score of 0.75. It is remarkable
that the scores in the neutral and positive category are again noticeably lower, with F1-scores
of 0.39 and 0.30. In total, the machine learning system reaches a fairly high accuracy score of
0.61. Furthermore, it is remarkable that the precision score of the positive category is nearly
three times higher than the recall score. This indicates that the machine learning system predicts
on the one hand very few positive stance labels, but that, on the other hand, these labels are in
many cases correct.
STANCE Negative Neutral Positive
Precision 𝑝 =246
(246 + 84 + 51)
𝑝 = 0.65
𝑝 =37
(33 + 37 + 9)
𝑝 = 0.47
𝑝 =10
(9 + 3 + 10)
𝑝 = 0.45
Recall
𝑟 =246
288
𝑟 = 0.85
𝑟 =37
124
𝑟 = 0.30
𝑟 =10
70
𝑟 = 0.14
F1-score
f = 0.75 f = 0.39 f = 0.30
Accuracy 𝑎 =246 + 37 + 10
482
𝑎 = 0,61
Table 11 - Precision, recall, F1-score and accuracy of stance labels
4.3.4.1 Impact of irony on the prediction of stance
To verify the impact of ironic language on the predictability of stance, Table 10 shows the
absolute numbers and percentages of irony in the Twitter corpus. In total 103 of 482 tweets
were manually labelled as ironic. The use of irony is fairly equally distributed between the
negative and neutral stance categories. It is, however, noticeable that the irony use in tweets
with positive stance is rather rare. In 26% and 16% of the corresponding negative and neutral
categories of true positives, the tweets contain irony. In addition, it can be stated that the irony
percentage in the false positives is just as high (26%) for tweets with negative stance and
somewhat higher (23%) for tweets with neutral stance.
Page 42 of 76
To interpret the meaning of these irony percentages, a closer look to the content of the falsely
labelled tweets is required. By considering merely the ironic tweets that were predicted falsely,
the following examples were retained.
(8) Can we solve #Brexit issues using the old "reset" method? Switch off, leave Europe
for 10 secs then plug ourselves bck in? Oh wait... oops?
(9) So, that was the dress rehearsal. Now that you Leavers have seen the effects of your
vote, would you like to try that again? #Brexit
(10) Churchill said, "Heroes fight like Greeks". Like a Greek i have to say that
"Heroes, vote like British!" #Brexit
In tweet (8) the author raises several rhetorical questions, a form of irony as defined by Karoui
et al. (2017). The classifier fails to pick up on this and returns a neutral stance label, whereas
the tweet was manually labelled as negative. The same goes for tweet (9), where the machine
learning system does not detect the ironic use of language and therefore decides to label the
stance as positive. The author, however, is against the outcome of the Brexit and jokingly
proposes to organize another referendum now the leave voters understand the negative
consequences of leaving the European Union. As a result, the tweet was manually annotated
with a negative stance label. In the last example (10), the author uses a metaphor and analogy
to express his positive attitude towards the Brexit. The system could not interpret the complex
nature of this tweet and assigned a negative label to it. Apart from tweet (8), (9) and (10), clear
examples of how irony influenced the detection of stance were rare.
4.3.4.2 Error analysis
As already established in chapter 4.3.2.2, it can be assumed that other factors influence stance
detection apart from irony presence. In this section, several examples of errors will be listed
and discussed.
(11) "protection!? Protection of what!? #TzeGermans!?" #Brexit #funfacts
(12) Proud of #Britain and @David_Cameron for doing the right thing today. #Brexit
#EUref #independence #freedom
Page 43 of 76
(13) Britons voted to strengthen their borders. Will you do the same in November?
#Brexit
(14) Your country is truly inspiring thank you #Brexit
(15) Man, that was so cool. #Brexit
(16) The #Brexit was necessary. The EU was turning into that evil corporation from
the Aliens franchise.
(17) On my room balcony posing with the freshly brexited EU flag. Giving it some
company. #iifa2016 #iifadiaries #brexit
The author’s stance in tweet (11) is negative. The content refers to one of the main arguments
of the Leave-voters in the Brexit referendum: leaving the EU is a form of protectionism and
equals ‘putting the UK first’. The author of tweet (11) uses a reference from the British movie
Snatch to point out that there is nothing to protect the UK from. The machine learning system
is not aware of this specific movie reference and judges that the stance is neutral. In tweets (12),
(13), (14) and (15) the trend that the classifier tends to overgenerate negative labels – just as
specified in the analysis of the SA results (cf. 4.3.2) – can be observed. The four tweets contain
positive stance and positive sentiment words. Nonetheless, the machine learning system decides
on a negative label. In the last example (16), the author appears to be in favour of the Brexit
outcome and against the EU as an institution. Therefore, the author’s stance towards the Brexit
is positive. It can be presumed that the classifier interpreted the stance as negative, as a
consequence of the use of ‘evil’. In the last tweet (17) with positive stance, there are no clear
sentiment words present, which results in a false attribution of a neutral stance label.
4.3.5 Comparison sentiment analysis and stance detection
In chapters 4.3.1, 4.3.2, 4.3.3 and 4.3.4 we analysed the results returned by the machine learning
system. We separately analysed the results of the sentiment analysis and stance detection, by
observing precision, recall, F1 and accuracy scores. In this section, we will focus on differences
Page 44 of 76
in performance and accordance between the prediction of sentiment and stance. Furthermore,
we will again consider the possible impact of irony use.
4.3.5.1 Comparison performance
In Table 12, all results of both the sentiment analysis and stance detection were listed. The first
and third column show the total number of tweets with negative, neutral or positive
sentiment/stance. The second and fourth column give an overview of how the machine learning
system attributed sentiment and stance labels. In addition, to compare the performance of our
system in terms of sentiment analysis and stance detection, Table 13 shows all precision, recall,
F1 and accuracy scores.
MANUALLY
ANNOTATED
SENTIMENT
TWEETS THAT ARE
AUTOMATICALLY
PREDICTED AS
MANUALLY
ANNOTATED
STANCE
TWEETS THAT ARE
AUTOMATICALLY
PREDICTED AS
NEGATIVE 254 NEGATIVE 199 NEGATIVE 288 NEGATIVE 246
NEUTRAL 40 NEUTRAL 33
POSITIVE 15 POSITIVE 9
NEUTRAL 144 NEGATIVE 81 NEUTRAL 124 NEGATIVE 84
NEUTRAL 54 NEUTRAL 37
POSITIVE 9 POSITIVE 3
POSITIVE 84 NEGATIVE 51 POSITIVE 70 NEGATIVE 51
NEUTRAL 23 NEUTRAL 9
POSITIVE 10 POSITIVE 10 Table 12 - Comparison results sentiment analysis and stance detection
SENTIMENT Negative Neutral Positive STANCE Negative Neutral Positive
Precision 𝑝 = 0.6 𝑝 = 0.46 𝑝 = 0.29
Precision 𝑝 = 0.65 𝑝 = 0.47 𝑝 = 0.45
Recall
𝑟 = 0.78 𝑟 = 0.38 𝑟 = 0.12 Recall
𝑟 = 0.85 𝑟 = 0.30 𝑟 = 0.14
F1-score
f = 0.69 f = 0.42 f = 0.21 F1-score
f = 0.75 f = 0.39 f = 0.30
Accuracy 𝑎 = 0.55 Accuracy 𝑎 = 0.61
Table 13 - Precision, recall, F1-score and accuracy of sentiment and stance labels
As can be drawn from Table 12 and 13, the classifier shows very similar results for the
prediction of both sentiment and stance. The negative and neutral precision scores are very
similar to one another. The positive precision score, however, is remarkably higher for the
prediction of stance labels (0.45) than for the prediction of sentiment labels (0.29). Overall, it
Page 45 of 76
is remarkable that the negative labels are predicted best, followed by the neutral category and
then the positive category. A possible explanation for this could be the smaller number of
neutral and positive tweets in our corpus, due to which the machine learning system had more
had more training examples to be trained on for the negative tweets.
In comparison, our classifier performs slightly better on stance detection, considering the
accuracy score of 0.55 for sentiment analysis and 0.61 for stance detection.
4.3.5.2 Comparison accordance and the influence of irony
In this section, we will compare the accordance of the manually assigned sentiment labels with
their stance labels (cf. Table 14) to the accordance of the automatically assigned sentiment
labels with their stance labels (cf. Table 15). In the fourth and fifth column, the number of ironic
tweets per sentiment/stance category was listed with their percentage.
SENTIMENT
PREDICTIONS
(classifier)
STANCE PREDICTIONS
(classifier)
ACCORDANCE IN
PERCENTAGES
NUMBER
OF IRONIC
TWEETS
IRONY
IN
PERCENTAGES
NEGATIVE 331 NEGATIVE 302 91% 64 21%
NEUTRAL 22 7% 4 18%
POSITIVE 7 2% 1 14%
NEUTRAL 117 NEGATIVE 60 51% 15 25%
NEUTRAL 50 43% 9 18%
POSITIVE 7 6% 2 29%
POSITIVE 34 NEGATIVE 19 56% 6 32%
NEUTRAL 7 20% 1 14%
POSITIVE 8 34% 1 13% Table 15 - Accordance of automatically assigned sentiment labels with their stance labels
From Table 14, it can be concluded that 86% of all tweets with negative sentiment also hold
negative stance. Then, for tweets with neutral sentiment, the correspondence with neutral stance
is somewhat lower, with a percentage of 62. Moreover, in 51% of all positive tweets the author’s
stance towards the Brexit is also positive. It is remarkable that negative tweets rarely hold
Table 14 - Accordance of manually assigned sentiment labels with their stance labels
SENTIMENT
PREDICTIONS
(manual annotation)
STANCE PREDICTIONS
(manual annotation) ACCORDANCE IN
PERCENTAGES NUMBER OF
IRONIC
TWEETS
IRONY
IN
PERCENTAGES
NEGATIVE 254 NEGATIVE 218 86% 47 22%
NEUTRAL 19 7% 3 16%
POSITIVE 17 7% 1 6%
NEUTRAL 144 NEGATIVE 45 31% 19 42%
NEUTRAL 89 62% 19 21%
POSITIVE 10 7% 0 0%
POSITIVE 84 NEGATIVE 25 30% 8 32%
NEUTRAL 16 19% 4 25%
POSITIVE 43 51% 2 5%
Page 46 of 76
neutral or positive stance, whereas differences between sentiment and stance are more frequent
in the neutral and positive category. A possible explanation for this could be the presence of
irony, especially for tweets with neutral/positive sentiment and negative stance. This means
that, at first glance, a certain tweet seems positive/neutral, however, after the interpretation of
irony use, it becomes clear that the author’s stance is actually negative. An example of this
could be: “If I had a time machine, I'd happily take a molesting from Farage, just to Yewtree
the cunt out of any political standing. #Brexit”. The sentiment in this tweet is positive because
of his enthusiasm expressed in the word ‘happily’. However, the author presumably does not
literally mean he would violently attack Nigel Farage, if he had a time machine. The author is,
in other words, ironically discussing his negative attitude/stance towards the politician.
In addition, Table 15 shows similar trends in the results of the machine learning system. Yet,
the results returned by the classifier do differ in terms of accordance in percentages, with the
exception of the negative category. In our gold standard, the majority of all tweets with neutral
sentiment hold neutral stance, whereas the machine learning system returns a higher percentage
of neutral tweets holding negative stance. The same goes for the largest manually annotated
category of positive tweets with positive stance, where the classifier finds more positive tweets
holding negative stance. Concerning the effect of irony in Table 15, it can be observed that the
highest irony percentages pop up in the neutral and positive tweets with negative sentiment.
Therefore, this could mean that our machine is able to interpret irony correctly to some extent.
However, from chapters 4.3.2.1 and 4.3.4.1 we learnt that in some cases the system fails to label
sentiment or stance correctly as a result of ironic language use.
5. CONCLUSION
With the emergence of Web 2.0 at the beginning of the 21st century, easy-accessible
microblogging platforms such as Facebook and Twitter have become omnipresent within the
digital landscape. Blogs, forums and social media websites allow users to easily share their
point of view, by means of blogposts, reviews, reactions and ratings. Due to the high amount
of feedback or criticism on, for instance, products, services or political ideas there was a call
for an automatic system that could gather a whole range of opinions on a certain topic. With
the help of SA, a business or organization can find out which sentiment (positive, negative or
neutral) a piece of text contains. As discussed by Pak and Paroubek (2009), political parties and
Page 47 of 76
politicians can gain a perception of how people view their programmes or how people see them
within the political landscape. In addition, SD aims at discovering the point of view expressed
by the author towards the subject being discussed. Nowadays, organisations consequently have
enough feedback at their disposal to examine how people’s views differ on a certain product,
service or policy. As a result, it is easier to detect the reason behind, for example, the success
of a certain campaign or the low sales figures of a certain product.
In this study, we attempted to contribute to the existing findings on automatic sentiment analysis
and stance detection on Twitter. Furthermore, we aimed to explore whether ironic language
influences the performance of machine learning systems. For these purposes, we built a Twitter
corpus consisting of 482 political tweets with the hashtag Brexit. The manually annotated
corpus was then compared to both the predicted sentiment and stance labels. Based on the
collected and analysed data and experimental results, we attempted to draw conclusions
concerning the predictability of sentiment and stance, as well as the impact of irony on the
performance of automatic systems.
Firstly, we observed that our Twitter corpus consisted mostly of negative sentiment (53%),
followed by tweets with neutral (30%) and positive sentiment (17%). Similarly, the machine
learning system also mostly predicted tweets with negative sentiment (69%), then tweets with
positive sentiment (24%) and lastly positive tweets (7%). The topic that was discussed the most
was the Brexit itself, presumably because all tweets already contained “#Brexit”. Furthermore,
negative labels were predominant in nearly every topic (such as ‘Brexit’, ‘USA’, ‘Trump’,
‘other’, …), which was a logical consequence, considering the high percentages of negative
tweets. We noticed that our machine learning system for sentiment analysis was rather biased
towards negative tweets, meaning that the classifier tended to overgenerate negative labels. It
was observed that negative labels were attributed falsely in 34% of the cases, whereas neutral
and positive labels were only assigned falsely in correspondingly 13 and 5 percent of the cases.
This could be a possible explanation for the high precision, recall and F1-scores within the
negative category and the rather low results in both the positive and neutral category. Overall,
the system scored 0.55 on accuracy which is, considering the relatively small amount of tweets
in our corpus, a fairly good score. What remains a challenge is the detection of implicit
sentiment (e.g. ‘feeling poor’ is a negative feeling), as was already stated by Van Hee (2017).
Page 48 of 76
Secondly, the distribution of the stance labels was relatively similar to the percentages of
negative, neutral and positive sentiment labels. The gold standard consisted mostly of tweets
with negative stance (60%), followed by tweets holding neutral (26%) and positive (5%) stance.
Our machine learning system predicted that 79% of all tweets contained negative stance, 16%
neutral stance and 5% positive stance. Regardless of the lower precision and recall scores in the
neutral and positive stance categories, our system scored 0.61 on accuracy, which is slightly
better than the score for sentiment analysis. It was noticeable that the system again tended to
overgenerate negative stance labels, which would explain the rising of the percentage of
negative labels predicted by the machine learning system with nearly a fifth. Furthermore, it
became clear in the analysis of the content of the tweets that the system failed to interpret
specific references to, for instance, movies correctly.
Regarding the accuracy results of our sentiment analysis (0.55) as well as stance detection
(0.61), our first hypothesis can be confirmed: the machine learning system delivers fairly
reliable results on a political Twitter corpus.
Thirdly, we considered the impact of the 103 ironic tweets in our corpus on the predictability
of both sentiment and stance. It appeared that true positives as well as false positives contained
similar irony percentages. As a result, it was rather hard to interpret whether irony influenced
the outcome of the sentiment analysis or stance detection, without evaluating the content of the
falsely predicted labels. Considering the content of all falsely labelled tweets, it could, however,
be concluded that in some cases irony was interpreted literally. In other words, irony did have
a certain negative effect on the performance of the machine learning system, but not in all cases.
As a result, our second hypothesis, which said that our machine learning system would not be
able to detect and interpret ironic language use correctly in most cases, cannot be confirmed
entirely.
In conclusion, it can be stated that a machine learning approach for sentiment analysis and
stance detection is already seemingly reliable. However, there is still room for improvement,
considering both accuracy scores are under 65%. Furthermore, we need to tackle the challenge
and indistinctness concerning ironic language in tweets. In chapter 6, suggestions on further
research will be formulated.
Page 49 of 76
6. LIMITATIONS AND FURTHER RESEARCH
Overall, we found that our system for sentiment analysis and stance detection scores fairly well,
however, there is still room for improvement. Therefore, we will discuss the limitations we
encountered during our research and we will provide some suggestions for further research in
this section.
We chose to build a corpus consisting of more or less 500 political tweets. Due to practical
limitations, the corpus was limited in size compared to other studies using Twitter corpora (cf.
Van Hee, 2017). This resulted in less reliable scores and percentages to interpret or to draw
conclusions on. Moreover, we had no insight into which sentiment words were or were not
detected by the machine learning system. Further research could, however, provide us with
more insight for the error analysis. Furthermore, it would be interesting to compare a machine
learning approach to a lexicon-based approach, to explore which approach generates the best
results.
In further research on sentiment analysis and stance detection on a Twitter corpus, it would also
be interesting to further explore the consequences of irony presence in political tweets. The
reason for this is twofold. Firstly, it remained relatively unclear in this research to which extent
irony influences the predictability of sentiment and/or stance. Secondly, the combination of
sentiment/stance on Twitter and irony has been rarely been investigated, which offers many
possibilities for improvement. Moreover, it would also be useful to perform irony detection on
the Twitter corpus itself, as was done by Van Hee (2017). In this study, we were not able to
automatically extract irony labels and thus, we could not analyse the effect of ironic language
thoroughly. For further research, it would be interesting to compare the results of, for example,
stance detection as well as irony detection.
Page 50 of 76
List of references
Bambrick, N. (24.06.2016). Support Vector Machines for dummies; A Simple Explanation.
Retrieved February 20, 2018 from http://blog.aylien.com
Barbieri, F., & Saggion, H. (2014). Modelling Irony in Twitter. In EACL (pp. 56–64).
Boertjes, E., Jamakovic A. & Raaijmakers S. (2011). Analyse van sociale media. STAtOR,
November, 3-4, 12, (pp. 41-47).
Cambria, E., Schuller, B., Xia, Y., & Havasi, C. (2013). New avenues in opinion mining and
sentiment analysis. IEEE Intelligent Systems, 28(2), 15–21.
Chang, C., Lin, C. (2016). LIBSVM: A Library for Support Vector Machines. Retrieved February
20, 2018 from https://www.csie.ntu.edu.tw/~cjlin/libsvm/
Dave, K., Lawrence S. & Pennock D.M. (2003). Mining the peanut gallery: Opinion extraction and
semantic classification of product reviews. In Proceedings of the 12th international conference
on World Wide Web(WWW), (pp. 519–528).
De Clercq, O., Lefever, E., Jacobs, G., Carpels, T., & Hoste, V. (2017). Towards an integrated
pipeline for aspect-based sentiment analysis in various domains. Proceedings of the 8th
Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media
Analysis (pp. 136–142). Presented at the EMNLP2017, Association for Computational
Linguistics.
Desmet, B., De Clercq, O., Van de Kauter, M., Schulz, S., Van Hee, C., & Hoste, V. (2014).
Taaltechnologie 2.0: sentimentanalyse en normalisatie. In Beschouwingen uit een talenhuis :
opstellen over onderwijs en onderzoek in de vakgroep Vertalen, Tolken en Communicatie
aangeboden aan Rita Godyns, (pp. 157–161).
Ghosh, A., Li, G., Veale, T., Rosso, P., Shutova, E., Barnden, J., & Reyes, A. (2015). Semeval-2015
task 11: Sentiment analysis of figurative language in twitter. In Proceedings of the 9th
International Workshop on Semantic Evaluation (SemEval 2015) (pp. 470–478).
Page 51 of 76
Karoui, J., Farah, B., Moriceau, V., Patti, V., Bosco, C., & Aussenac-Gilles, N. (2017). Exploring
the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study.
In Proceedings of the 15th Conference of the European Chapter of the Association for
Computational Linguistics: Volume 1, Long Papers (Vol. 1, pp. 262–272).
Khan, A. Z., Atique, M., & Thakare, V. M. (2015). Combining lexicon-based and learning-based
methods for Twitter sentiment analysis. International Journal of Electronics, Communication
and Soft Computing Science & Engineering (IJECSCSE), 89.
Kolkur, S., Dantal, G., & Mahe, R. (2015). Study of different levels for sentiment analysis.
International Journal of Current Engineering and Technology, 5(2), 768Ű770.
Kreuz, R. J., & Roberts, R. M. (1993). On satire and parody: The importance of being
ironic. Metaphor and Symbol, 8(2), 97-109.
Kulshrestha, J., Kooti, F., Nikravesh, A., & Gummadi, P. K. (2012, June). Geographic Dissection of
the Twitter Network. In ICWSM.
Kumar, A., & Sebastian, T. M. (2012). Sentiment analysis on twitter. IJCSI International Journal of
Computer Science Issues, 9(3), 372–378.
Liu, B. (2012). Sentiment Analysis and Opinion Mining. Synthesis lectures on human language
technologies, 5(1), 1-167.
Mandya, A., Siddharthan, A., & Wyner, A. (2016). Scrutable feature sets for stance classification.
ACL 2016, 60.
Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X.-D., & Cherry, C. (2016). SemEval-2016 Task
6: Detecting Stance in Tweets. In SemEval@ NAACL-HLT (pp. 31–41). Retrieved from
http://www.aclweb.org/anthology/S/S16/S16-1003.pdf
Nasukawa, T., & Yi, J. (2003). Sentiment analysis: Capturing favorability using natural language
processing. In Proceedings of the 2nd international conference on Knowledge capture (pp. 70-
77). ACM.
Page 52 of 76
Nielsen, F. (2011). IMM Publications: AFINN. Retrieved February 20, 2018 from
http://www2.imm.dtu.dk/
Pak, A., & Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. In
LREc (Vol. 10).
Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends. In
Information Retrieval, 2(1–2), 1-135
Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs Up?: Sentiment Classification Using
Machine Learning Techniques. In Proceedings of the ACL-02 Conference on Empirical
Methods in Natural Language Processing - Volume 10 (pp. 79–86). Stroudsburg, PA, USA:
Association for Computational Linguistics. doi: 10.3115/1118693.1118704
Pavlopoulos, I. (2014). Aspect based sentiment analysis. Athens University of Economics and
Business.
Pontiki, M., Galanis, D., Papageorgiou, H., Androutsopoulos, I., Manandhar, S., AL-Smadi, M., …
others. (2016). SemEval-2016 task 5: Aspect based sentiment analysis. In ProWorkshop on
Semantic Evaluation (SemEval-2016) (pp. 19–30). Association for Computational Linguistics.
Retrieved from https://biblio.ugent.be/publication/8131987
QuantStart Team. (12.09.2014). Support Vector Machines: A Guide for Beginners. Retrieved
February 20, 2018 from https://www.quantstart.com/
Reyes, A., Rosso, P., & Buscaldi, D. (2012). From humor recognition to irony detection: The
figurative language of social media. Data & Knowledge Engineering, 74, 1–12.
Reyes, A., Rosso, P., & Veale, T. (2013). A multidimensional approach for detecting irony in twitter.
Language Resources and Evaluation, 47(1), 239–268.
Shoeb, M., & Ahmed, J. (2017). Sentiment Analysis and Classification of Tweets Using Data
Mining. Work, 4(12).
Page 53 of 76
Taboada, M. (2016). Sentiment Analysis: An Overview from Linguistics. Annual Review of
Linguistics, 2. doi:10.1146-011415-040518
Tumasjan, A., Sprenger, T. O., Sandner, P. G., & Welpe, I. M. (2010). Predicting elections with
twitter: What 140 characters reveal about political sentiment. Icwsm, 10(1), 178–185.
Van de Kauter, M. (2013). LeTs preprocess: the multilingual LT3 linguistic preprocessing toolkit.
Van Hee, C. (2017). Can machines sense irony? : exploring automatic irony detection on social
media. Ghent University. Faculty of Arts and Philosophy, Ghent, Belgium.
Van Hee, C., Lefever, E., & Hoste, V. (2015). LT3: sentiment analysis of figurative tweets: piece of
cake #NotReally. In SemEval : 9th International Workshop on Semantic Evaluations at Naacl
2015, Proceedings (pp. 684–688). Association for Computational Linguistics. Retrieved from
http://hdl.handle.net/1854/LU-6827146
Van Hee, C., Van de Kauter, M., De Clercq, O., Lefever, E., & Hoste, V. (2014). LT3: sentiment
classification in user-generated content using a rich feature set. In Proceedings of the 8th
International Workshop on Semantic Evaluation (pp. 406–410). Association for Computational
Linguistics. Retrieved from http://hdl.handle.net/1854/LU-5684543
Wang, P.-Y. A. (2013). # Irony or# Sarcasm—A Quantitative and Qualitative Study Based on
Twitter. In Proceedings of the 27th Pacific Asia Conference on Language, Information, and
Computation (PACLIC 27) (pp. 349–356).
Zhang, L., Ghosh, R., Dekhil, M., Hsu, M., & Lius, B. (2011). Combining Lexicon-based and
Learning-based Methods for Twitter Sentiment Analysis. In Hewlett-Packard
Labs Technical Report HPL-2011-89.
Page 54 of 76
APPENDIX 1
Brexit - Brexit
- Consequence Brexit
- Ignorance about Brexit
- Joke on Brexit
- Link to article on Brexit
- Link to video on Brexit
- Second referendum
- Voters
Celebrities - Angela Merkel
- Boris Johnson
- Boris Johnson + link to article
- Celebrities
- Celebrities & link to article on Brexit
- Celebrities (Sarah Palin)
- David Cameron
- Hilary Clinton
- Lindsey Lohan
- Nicola Sturgeon
- Obama
- Thatcher
Economy - Economy
- Economy
- Voters & economy
EU - EU
- EU & Brexit
- EU & NATO
- EU & UN
Other - Article
- British housing
- Citizenship
- British politics
- Consequences & voters
- Democracy
- Education
Page 55 of 76
- Football
- German Foreign Office
- Greece
- Housing
- Immigration
- International politics
- Japan
- Joke
- Leftists
- Link to an article
- Monarchy
- Other
- Personal information
- Petition
- Press
- Racism
- Refugees
- Science
- Scotland and Ireland
- Sore losers
- Texas
- UN
- US housing
- Work opportunities
Scottish referendum - Economy & Scottish referendum
- Scottish referendum
- Scottish referendum + link to article
Trump - Brexit & Donald Trump
- Brexit & Trump
- Donald Trump
- Donald Trump + Texas
- Insults about Donald Trump
- Trump
- USA & Trump
USA - America
- American elections
Page 56 of 76
- Brexit & US
- US
- USA
Page 57 of 76
APPENDIX 2
Tweet senti
ment
topic/aspec
t
stan
ce
tow
ards
the
Bre
xit
iro
ny
Senti
ment
predic
tions
Stanc
e
predic
tions
$2.7 TRILLION lost on global markets after
@RupertMurdoch has his #Brexit dreams come true. People
will die from impacts of such losses.
negat
ive economy
nega
tive 2 2
@sorrelita "protection!? Protection of what!?
#TzeGermans!?" #Brexit #funfacts
neutr
al other
nega
tive 2 3
Final #Brexit tally is in: 48% Sense and Sensibility, 52%
Pride and Prejudice.
neutr
al other
nega
tive 2 2
Do we even care? – #Brexit aftermath
http://snowcalmth.online/dowecare/ Overview: - Blogposts
delayed. - The Forgotten #Youth - Do we know the #EU?
negat
ive other
nega
tive 3 3
#Brexit, Monty Python & Silly Walks. @NewYorker neutr
al other
nega
tive 2 3
Proud of #Britain and @David_Cameron for doing the
right thing today. #Brexit #EUref #independence #freedom
positi
ve Brexit
posit
ive 2 2
@stuartpstevens @AshleyRParker I didn't know about
#brexit either just some gun control stunt
negat
ive Brexit
neut
ral 2 2
"I told y'all to vote REMAIN tho" #Brexit negat
ive Brexit
posit
ive 3 3
"The World Turned Upside Down." Not necessarily the
song I wanted to be humming this morning. #Brexit
negat
ive Brexit
nega
tive 1 1
They want to put all of US to the back of the bus to their
global masters. We say...piss off! #Brexit #Trump2016
negat
ive USA
posit
ive 2 1
said of #Brexit, while in Scotland IN SCOTLAND!!!!! negat
ive Trump
nega
tive x 1 2
Why are #liberal #democrats all over TV crying about
#Brexit and the markets? Didnt you people want the big banks
to lose money and power?
negat
ive economy
nega
tive x 2 2
Boris Johnson goes from court jester to crown prince after
#Brexit win http://bloom.bg/28RCZLw
neutr
al
celebrities/p
oliticians
neut
ral 3 3
Love being in Sweden, Bruce in every bar #priceyoupay
#sweden #Brexit
positi
ve other
neut
ral 2 2
Breaking!!! #UK votes to ease epcot!! #BrexitVote
#Brexit #BrexitHumor #tcot #RedNationRising
neutr
al Brexit
neut
ral
1 3
Lindsay Lohan fumes over #Brexit, Elizabeth Hurley
sleeps soundly
neutr
al
celebrities/p
oliticians
neut
ral
2 2
@torontodan As the final lines of the old Presbyterian
joke go: "Lord, Lord, we didna ken". "Weel, Weel, ye ken
noo".#Brexit
negat
ive Brexit
nega
tive x 2 2
Why #Brexit is terrible for UK science, in one map
http://www.economist.com/news/britain/21699504-most-
scientists-want-stay-eu-european-experiment …
negat
ive Brexit
nega
tive 2 2
.@piersmorgan - I always thought #Brexit was that good
dump you take after morning coffee. TOTALLY confused. Pls
call me.
negat
ive Brexit
nega
tive x 2 2
Ahead on New York Tonight: Leaders react to #Brexit.
@POTUS designates @TheStonewallNYC as nat'l monument,
and a preview of #NYCPrideMarch
neutr
al Brexit
neut
ral 2 2
Calls for a second Scottish independence referendum
grow louder after #Brexit http://econ.st/28XxXhn
neutr
al
Scottish
referendum
nega
tive 3 3
The nightmare of the #EU & #UN Elites is for these two
nations to rise up and say ENOUGH! #Brexit #America
#Britain
negat
ive EU
nega
tive 2 2
I leave Twitter for a week and #Brexit happens. neutr
al Brexit
neut
ral 3 3
Page 58 of 76
You know what's "bizarre"? Media folks who see the
#Brexit vote & take virtually no relevant lesson from it.
negat
ive Brexit
nega
tive 2 2
Watch Christiane Amanpour Get ANGRY That Britain
ALLOWED The People To Vote On #Brexit
negat
ive Brexit
nega
tive 2 2
.@LizClaman: “The losses on paper are now tallying
$900 billion on the U.S. stock market.” #Greta #Brexit
negat
ive economy
nega
tive 2 2
No: History will show that #brexit Is good for nobody negat
ive Brexit
nega
tive 3 3
Take Note! Here in the States this would be the equivelant
of “I thought it wud be funny to vote for Trump" #Brexit
negat
ive USA
nega
tive 2 2
Shall we start preparing EU shores for an influx of British
refugees? #brexit
negat
ive Brexit
nega
tive x 3 2
I'm going to go with #brexit damage control for $1000
Alex!
positi
ve economy
nega
tive 2 2
SG #NATO Sees Unifying Role as #UK #Brexit Shakes
#Allies:#Britain’s vote to leave #EU leaves Europe more
fragmented.
neutr
al EU
nega
tive 2 2
Lindsay Lohan passionately expressed her stance against
#Brexit in now-deleted tweets http://peoplem.ag/ullfemm
neutr
al
celebrities/p
oliticians
neut
ral 1 3
Celebrating #Brexit with the British Players in beautiful
Kensington, MD.
positi
ve Brexit
posit
ive 3 2
Odd how some on the right are in despair over #Brexit.
Socialist globalization is no way to run a planet, people.
negat
ive Brexit
nega
tive 2 2
Poor folks voting for #Brexit is the equivalent of a Turkey
voting for Thanksgiving. White Nationalists can never be
accused of rationality.
negat
ive Brexit
nega
tive x 2 2
Oh, God, they're giving the keys for the Tridents to BoJo
the Clown... #Brexit
negat
ive
celebrities/p
oliticians
nega
tive 2 2
Britons voted to strengthen their borders. Will you do the
same in November? #Brexit
positi
ve USA
posit
ive 3 2
Wales should have been more careful what it wished for.
It's going to be given it. #Brexit
negat
ive Brexit
nega
tive 3 2
And suddenly the birds are singing.....still glued to the TV
though #Brexit
positi
ve Brexit
posit
ive 3 2
The worst has yet to come #sarahpalin #brexit
#exitstupidity
negat
ive Brexit
nega
tive
2 2
#Blairites using #Brexit to (yet again) try & unseat
#Corbyn proof of their lack of allegiance to Labour.
negat
ive
celebrities/p
oliticians
neut
ral 2 2
Hey United Kingdom imma let you finish but America
had one of the greatest #Brexit's of ALL TIME
positi
ve USA
neut
ral x 1 3
Good branding: #Brexit, how "attractive" is the name! +
Good hype: Thanks to #SocialMedia. + Intense emotion: Fear
= Results = #Marketing101
positi
ve Brexit
posit
ive x 2 2
Really interesting piece on the #Brexit where the fragility
of masculinity surfaces again.
positi
ve Brexit
neut
ral 2 2
.@jasonrileywsj: EU needed Britain more than Britain
needed the EU -OTR #greta #Brexit #PoliticalPanel
@FoxNews
neutr
al EU
posit
ive 2 2
Don’t think #brexit is a big deal? Here’s a chart. negat
ive Brexit
nega
tive 2 3
Everyone should be worried about a Trump #Drumpf
presidency! Now #Brexit! World heading in an interesting
direction
negat
ive Trump
nega
tive 2 2
#Brexit threatens damage to U.S.-UK ties, could
embolden Russia's Putin/@mattspetalnick @yarabayoumy
negat
ive Brexit
nega
tive 2 2
#Academics fear new #Brexit – a brain exit – after
#referendum vote
http://www.independent.co.uk/news/science/brain-drain-brexit-
universities-science-academics-referendum-eu-
a7100266.html … #EURefResults
negat
ive
Brexit
nega
tive 3 3
The #Brexit was necessary. The EU was turning into that
evil corporation from the Aliens franchise.
positi
ve EU
posit
ive 2 2
GBP weakens, world markets fall, housebuilders shares
drop 25%. Scottish referendum 2.0? Youth vote ignored.
Cameron resigns. #Brexit #Day1
negat
ive Brexit
nega
tive 2 2
Page 59 of 76
The @realDonaldTrump right again. #Brexit #VoteLeave
#UKIP #UK #MakeAmericaGreatAgain #TrumpTrain
positi
ve Trump
posit
ive 3 1
With all the hysteria and over the top hyperbole over
Brexit, maybe we should all just take a deep breath and relax
for a few days. #Brexit
neutr
al other
neut
ral 3 2
Lonng #Brexit day done, let's get back to #IREvFRA!!
#Remain #EURO2016
positi
ve other
posit
ive 2 2
Brexit vote: Anger in the bedroom, joy on the streets
http://cnn.it/28V6bCD #Brexit #seeEUlater
neutr
al Brexit
neut
ral x 2 3
Trump/Sanders #Brexit philosophy: Seize corps overseas
ops & ban foreign sales. HUGE govt! @SMShow
@SueinRockville
neutr
al USA
neut
ral 2 2
Can Nicola Sturgeon get a Faroes-style opt out from
#Brexit? And might that persuade Scots we r ready 4
#indyref2?
neutr
al Brexit
neut
ral x 3 2
On my room balcony posing with the freshly brexited EU
flag. Giving it some company. #iifa2016 #iifadiaries #brexit
positi
ve Brexit
posit
ive 2 3
Do those who claim it is "stupid" to propose Esperanto as
official for EU after #Brexit even know anything about the
language?
negat
ive other
neut
ral x 2 2
Still so sad about #Brexit. What is this dark, absurd future
being carved out for the world?
negat
ive Brexit
nega
tive 2 2
If I had a time machine, I'd happily take a molesting from
Farage, just to Yewtree the cunt out of any political standing.
#Brexit
negat
ive Brexit
nega
tive x 1 2
@charlescwcooke and I disagree on #Brexit. But I love
his British understatement.
negat
ive Brexit
nega
tive 1 2
your country is truly inspiring thank you #Brexit positi
ve Brexit
posit
ive 2 2
A lot of the #Glastonbury audience are wearing either
#Hibs tops, #sunglasses or #SantaHats - I am confused in this
post-#brexit world.
negat
ive other
nega
tive 2 2
Briton on FB claims his country was "raped" - you loose a
referendum and your country was raped? #brexit #democracy
negat
ive Brexit
posit
ive
2 2
Great Britain secedes from the European Union. Millions
in the EU now looking for real jobs. #Brexit
neutr
al EU
nega
tive
2 2
Massive props to David Cameron for giving Poms chance
to decide own fate, honourably stepping aside when result
against him. #brexit
positi
ve celebrities/p
oliticians
neut
ral 2 2
#Brexit no, Corbyn has a great deal to answer for as do
many others, its not just about the Tories however unpopular
that view is.
negat
ive celebrities/p
oliticians
nega
tive 2 2
The Working Classes will be the first to be shunted! So
short sighted! #Brexit #shouldhavegonetospecsavers
#universityofJeremyKyle
negat
ive Brexit
nega
tive 2 2
business: Boris Johnson goes from court jester to crown
prince after #Brexit win http://bloom.bg/28RCZLw
neutr
al
celebrities/p
oliticians
neut
ral 3 3
I never thought of Britain as being European anyway.
#Brexit
negat
ive EU
posit
ive 2 2
Instead of posting hilarious gifs maybe the #remain side
should start thinking about the future of this country outside of
the EU #Brexit
negat
ive Brexit
nega
tive 2 2
"World's 400 Richest Lose $127 Billion" It's happening!
#Brexit #LeaveWins #FirstHubrisThenNemesis
http://bloom.bg/28TnoIm
neutr
al economy
neut
ral 3 2
<em>The Atlantic</em> Daily: The Great British Break-
Off #brexit http://brexitwhatnext.com/2016/06/emthe-
atlanticem-daily-the-great-british-break-off/ …
neutr
al Brexit
neut
ral x 3 3
#Trump reaction to #Brexit: 1) See, they like my wall
plan too, and 2) I'll make $$ off the tanking pound. #prick
negat
ive Trump
nega
tive 2 2
Only 72% voter turnout to a decision that literally
changed the entire economy. #WakeUpPeople #Brexit
negat
ive economy
nega
tive 2 2
Do you think Boris Johnson will become the next Prime
Minister of Great Britain?
http://americansdecide.com/topic/do-you-think-boris-johnson-
neutr
al celebrities/p
oliticians
neut
ral 2 2
Page 60 of 76
will-become-the-next-prime-minister-of-great-britain/ …
#Brexit
Is this going to effect my chances of getting into
Hogwarts? #Brexit
neutr
al other
nega
tive x 2 2
This is one factor of the #Brexit vote. Lot more
complications than this.
negat
ive Brexit
nega
tive 2 2
' See EU Later 'was my best headline on #Brexit positi
ve other
nega
tive 2 3
#Calls For #Texas #Independence #Surge In #Wake Of
#Brexit #Vote - http://www.angrysummit.com/calls-for-texas-
independence-surge-in-wake-of-brexit-vote …
neutr
al other
neut
ral 3 3
Even his wife doesn't believe him...just look at her
expression. This man is a pariah #DavidCameron #Brexit
negat
ive
celebrities/p
oliticians
nega
tive 2 2
Assume everything said by politicians lawyers &
corporate moguls is a lie until U see credible proof. #Brexit
negat
ive Brexit
nega
tive 2 2
Not liking this because it's exactly what I feared #Brexit
lot have led everyone into #neverneverland OMG
negat
ive Brexit
nega
tive x 2 2
Interesting how we act on uncertainty when things are
under control. Panic creates crisis.. instead of keeping a cool
head #Brexit aftermath
negat
ive Brexit
nega
tive 2 2
The UK mucked up big time and the UKIP still wants
access to the EU market? Lmao what a joke #Brexit
#EURefResults #UKreferendum
negat
ive economy nega
tive
2 2
Man, that was so cool. #Brexit positi
ve Brexit
posit
ive
2 2
Putting himself 1st, Trump says #Brexit will help HIS
resort: “When the down pound goes down, more people are
coming to Turnberry, frankly" - Why…because now they have
to swim the #EnglishChannel to invade Britain?
negat
ive
Trump
nega
tive x 2 2
Sooo like everyone today, I logged in to check my 401k
because #Brexit...expecting it to be but is..anyone want to
explain?
neutr
al economy
neut
ral 2 2
Ya fucked up #Brexit negat
ive Brexit
nega
tive 2 2
HAHAHA! Everyone's trying to move here (Canada),
now! #Brexit #BrexitOrNot #BelieveItOrNot
positi
ve Brexit
nega
tive 2 2
This. #Brexit neutr
al Brexit
neut
ral 2 2
Well #Brexit happened. Here's a look at the possible
impacts on UK #RealEstate.
http://www.forbes.com/sites/carlapassino/2016/06/24/will-the-
uks-real-estate-sector-survive-brexit/#1427abb22120 …
#pound
neutr
al
Brexit
neut
ral 3 2
#Brexit should be a wake-up call to US #liberals: don’t
assume Drumpf will lose
http://www.vox.com/2016/6/24/12023816/brexit-donald-
trump-
winning?utm_campaign=vox&utm_content=article%3Afixed
&utm_medium=social&utm_source=twitter … via
@voxdotcom #Worried
negat
ive Trump
nega
tive
2 2
Hozier calls #Brexit a "massive betrayal": "My heart
breaks" http://blbrd.cm/iAA1ss pic.twit...
http://bit.ly/296vVcl #ShowTime
negat
ive
celebrities/p
oliticians neut
ral
3 3
What does #Brexit mean to the San Francisco housing
market? Read more from @PacUnion Chief Economist Selma
Hepp:
neutr
al Brexit neut
ral
2 2
Proud to be #Brexit! Proud to stand alone in a world
where most are too scared to be alone, to have their own
opinion. Proud to me! #Ukref
positi
ve Brexit posit
ive 2 2
Repeated refrain re #Brexit -Elections have consequences
-remember that in November folks #NeverTrump
#Election2016
negat
ive USA nega
tive 2 2
If you care about our future join 450,000 people
petitioning parliament for a 2nd referendum
http://www.independent.co.uk/news/uk/brexit-petition-for-
negat
ive Brexit nega
tive 2 2
Page 61 of 76
second-eu-referendum-so-popular-the-government-sites-
crashing-a7099996.html# … #Brexit
"Drunk Shakespeare, probably the only proper activity
after #Brexit https://www.instagram.com/p/BHDnj20jXxO/
negat
ive Brexit
nega
tive x 2 2
Can anyone tell me what's the status of eu member state
citizens now residing in the Uk? Illegal? Visa? #Brexit
#BrexitVote #Lexit #Leave
neutr
al Brexit nega
tive x 3 2
This #Brexit thing has me really worked up. It doesn't
bode well for the U.S. staying in the EU.
negat
ive Brexit
nega
tive 2 2
After #Brexit, another EU is possible with UK, Norway
and Switzerland.
neutr
al Brexit
nega
tive x 3 3
I think is the natural progression of human society to
become more integrated as time advances. #Brexit is the old
world fighting back.
negat
ive Brexit nega
tive 2 2
Why the surprise over #Brexit? This is the same country
that threw Churchill out of office after he pulled their nuts out
of the fire in WW2
neutr
al Brexit nega
tive 2 3
Secede and keep seceding, don't stop until you get to the
individual! #Brexit
negat
ive Brexit
nega
tive x 2 2
If u are rich & white- yup! Here is #Brexit promise
walked back hours after count
negat
ive Brexit
nega
tive 2 2
Brits will be poorer because #brexit but don't have to
weigh bananas with #metric system. A wise people, indeed.
@ilduce2016 @billmaher
negat
ive Brexit nega
tive 2 2
#Brexit: Do you suspect that the new negotiated treaties
will replicate membership in the European Union?
#worldismovingtoofastdepartment
negat
ive Brexit nega
tive 2 2
Traditionalist Catholic blog: The Filioque Clause.
http://www.stuart-filioque.blogspot.com #PopeFrancis #Brexit
neutr
al other
neut
ral 2 2
This is not like Idiocracy. In Idiocracy, once they found a
smart person, they made him fix their problems. #Brexit
negat
ive Brexit
nega
tive x 2 2
Nice attempt at making shit up! #Brexit #abc7chicago
#iteam #millennials ??
negat
ive Brexit
nega
tive 3 2
We won't win Eurovision for 69 years #EUref #Brexit negat
ive Brexit
nega
tive x 2 2
Beneath the cross of Jesus, His family is my own. #Brexit
#EUref
neutr
al other
neut
ral 2 2
Britons seek to 'move to Canada' after #Brexit vote neutr
al Brexit
nega
tive 2 2
A very good cereal served at my amazing property in
Turnberry! #Brexit of Champions, just like me! Enjoy!
positi
ve other
posit
ive 2 2
I'd like bier, croissant and wusrt please. What's the tinned
stuff? spam? #Brexit #EURefResults #WhatHaveWeDone
negat
ive Brexit
nega
tive 3 2
I've never seen Americans talk about Britain on my
Twitter feed before. And they're all taking the piss #Brexit
negat
ive Brexit
nega
tive 2 2
I did speak out on the positives of a sensible #Brexit based on
democratic process. But this is just populist bollocks.
negat
ive Brexit
nega
tive 2 2
Attention fellow Scots!!! It's ok!! I've had an idea!! We
can build a wall... #Brexit #remain
#MakeDonaldDrumpfAgain
negat
ive other nega
tive 2 2
I dare not go to YouTube for #Brexit videos. Can you
imagine the dross?
negat
ive Brexit
nega
tive 2 2
Learning some great new swears, thanks Scotland!
#Brexit
positi
ve other
nega
tive 1 2
#BrExit is exercise of the most important check on elite
mismanagement - the peoples' power to vote for HORRIBLE
IDEAS.
negat
ive Brexit nega
tive 2 2
#Regrexit: Speculation grows on the options for #Brexit
actually NOT happening.
https://waitingfortax.com/2016/06/24/when-i-say-no-i-mean-
maybe/ …
negat
ive Brexit
neut
ral 2 2
The U.K. could really use a more robust system of
excheques & balances #Brexit #BrexitVote
negat
ive Brexit
posit
ive 3 3
#Brexit Is Sending Markets Diving. #Twitter Could Be
Making It Worse http://dlvr.it/Lf6Zcl #Wired
negat
ive economy
nega
tive 2 3
Page 62 of 76
Brought some humor to the situation @camilluddington
#Brexit
positi
ve other
nega
tive 2 2
I've changed my mind. #Brexit is good. positi
ve Brexit
posit
ive 3 3
Brought some humor to the situation #BrexitVote #Brexit
#BrexitHumor #tcot #RedNationRising
positi
ve other
nega
tive 3 2
Okay can someone explain #brexit to me neutr
al Brexit
neut
ral 2 3
The French now want to move the Calais 'jungle' migrant
camp to British soil after #BREXIT
neutr
al Brexit
nega
tive 2 3
Patsy & Edina totally voted Remain. #Brexit negat
ive Brexit
nega
tive x
3 1
Mass referendums at their best Brits don't know what they
voted for #Brexit #EU #EuropeanUnion
#DavidCameronResigns
negat
ive Brexit nega
tive
2 2
#Brexit: the day rational choice theory blew up into
thousand pieces
negat
ive Brexit
nega
tive 3 2
Churchill said, "Heroes fight like Greeks". Like a Greek i
have to say that "Heroes, vote like British!" #Brexit
positi
ve Brexit
posit
ive
2 2
Can we have a redo? Where is the reset button? #Brexit negat
ive Brexit
nega
tive
2 2
#Brexit – The New Modern-Nationalism is #Global
#Governance - http://www.angrysummit.com/brexit-the-new-
modern-nationalism-is-global-governance …
#ModernNationalism
neutr
al Brexit
neut
ral 2 3
WaPo: #Brexit vote sends a message to politicians
everywhere: It can happen here
neutr
al Brexit
nega
tive
2 2
Now keep the promise of £350m a week for our #NHS -
Sign the petition: #EuRef #Leave #Brexit
https://you.38degrees.org.uk/petitions/invest-ps350-million-
saved-from-eu-in-nhs-by-2018?bucket=fb&source=twitter-
share-button … via @38_degrees
neutr
al Brexit
posit
ive
3 1
A big day with #Brexit, but I made history too when I
bent over to tie my shoe at the gym and a guy rushed over to
ask if I was OK. #only42!
negat
ive other nega
tive x 1 2
The pound goes down and so do stocks #Brexit negat
ive economy
nega
tive 3 3
Last chance to vote: Is #Brexit good for
@realDonaldTrump? Tweet YES OR NO using #greta
@FoxNews
neutr
al USA neut
ral 2 2
Leftists are freaking out over the #Brexit. Why, because
the people finally rejected tyranny? Just proves: liberals =
tyrants. #tcot
negat
ive Brexit posit
ive 2 2
#Brexit #Leave Please, welcome a #Britishrefugee neutr
al Brexit
nega
tive 2 2
Congrats to the UK for #Brexit positi
ve Brexit
posit
ive 2 2
Last in first Out #brexit #uk #eu #eng neutr
al Brexit
neut
ral 3 3
@JaneNormanInt Need to make my order now when it is
still possible before #Brexit. Any plans to move your office to
#EU? #onlineordering
neutr
al other neut
ral 3 2
ISIS takes credit for every terrible thing that happens on
earth, but even they're saying today "don't hang that #Brexit
crap on us!"
negat
ive Brexit nega
tive x 2 2
#Brexit Well, that required an active stupidity that rivals negat
ive Brexit
nega
tive x 2 2
#Brexit’s uncertainty in Europe will ripple back to Central
Texas http://atxne.ws/28T4wvL
negat
ive other
nega
tive 3 3
"Migration isn't the underlying cause of the thrust towards
#Brexit. Austerity is." - @yanisvaroufakis
negat
ive Brexit
nega
tive 2 2
UK want to leave the republic...aaahm eu? First thing in
my mind is a clone army #StarWars #Brexit #justkidding #sad
but that's #democracy
neutr
al Brexit neut
ral x 3 1
#Brexit: Up until midnight last night #voteremain was
leading on social media: http://brnw.ch/28Tnrr1
neutr
al Brexit
neut
ral 2 2
Page 63 of 76
Beginning of the end for the European Union: Best précis
of impact of #Brexit I've read to date. #corpgov #strategy
negat
ive Brexit
nega
tive 2 2
Hillary Clinton urges 'experienced leadership' after
#Brexit from #EU http://goo.gl/fb/0eksvp #europe
#europeanunion
neutr
al Brexit neut
ral 3 2
#Brexit could break up #EU and #NATO, prevent World
War III: Paul Craig Roberts http://goo.gl/fb/Ihi2EQ #europe
negat
ive Brexit
nega
tive
3 2
Britain: Let's grab a pint. EU: No thanks. I don't drink
during the day. #Brexit
negat
ive Brexit
nega
tive x
2 2
#Brexit, the political equivalent of cat videos negat
ive Brexit
nega
tive x 2 2
JPMorgan staff memo from Jamie Dimon, others about
Brexit referendum vote #Brexit #JPMorgan #UKref
neutr
al Brexit
neut
ral 2 2
Scotland, Wales, & London voted to #Remain, everyone
else voted for #Brexit. #Texas wants to #Secede. Can we trade
Texas for those first 3?
positi
ve Brexit posit
ive x 3 2
sooooooo i still have over £40 that i never exchanged for
dollars. i feel less guilty about that now. #Brexit
positi
ve other
neut
ral 2 3
Hillary Clinton urges 'experienced leadership' after
#Brexit from #EU http://goo.gl/fb/aoXo3G #europe
#europeanunion
neutr
al Brexit neut
ral 3 2
#Brexit could break up #EU and #NATO, prevent World
War III: Paul Craig Roberts
http://goo.gl/fb/qkMCWm #europe
neutr
al Brexit neut
ral 3 2
MSM treats #Brexit as Europe's demise.History might say
it set in motion needed revisions of Social Contract&economic
machinery of Europe
positi
ve Brexit posit
ive 3 2
President Vladimir Putin says #Russia has 'never
interfered' in #Brexit http://goo.gl/fb/Dq5PGX #eu #europe
neutr
al Brexit
neut
ral 3 3
Top Google search in Britain AFTER the #Brexit vote
was "What is the EU?" After the vote? #Feckin morons need a
monarch.
negat
ive other nega
tive
3 2
Also strenghtened by the renovated and expanded flexible
credit line with the IMF #Brexit #Mexico #PressRelease
neutr
al Brexit
neut
ral
1 2
President Vladimir Putin says #Russia has 'never
interfered' in #Brexit http://goo.gl/fb/tJXPni #eu #europe
neutr
al Brexit
neut
ral 3 3
Boris Johnson goes from court jester to crown prince after
#Brexit win http://bloom.bg/28RCZLw
neutr
al
celebrities/p
oliticians
neut
ral 3 3
I h8 the phrase "take back our country," whether it's used
4 the US or the UK bc it's fear-mongering by spreading hate 4
foreigners #Brexit
negat
ive Brexit nega
tive 2 2
My vote was to be free of unelected EU commissioners
passing laws that our country has no say in. Glad to be out,
they need us more. #Brexit
positi
ve Brexit posit
ive 2 2
@truthout It hasn't even been 24 hours and you're judging
the outcome? Britain hasn't even officially the left EU yet.
neutr
al Brexit
posit
ive 2 2
Hillary Clinton represents the crony capitalist kleptocracy
the author identifies as responsible for #BREXIT. DOA.
negat
ive
celebrities/p
oliticians
nega
tive 2 2
The latest The Sciarra Stefano Daily!
http://paper.li/Colonnasciarra/1334314750?edition_id=998632
10-3a67-11e6-92d0-0cc47a0d1605 … Thanks to @rpinci
@VentagliP @Surfiniae #brexit #business
neutr
al other
neut
ral 3 3
True elites want to run their own lives and own countries
not be told by central government what to do. #Brexit
positi
ve Brexit
neut
ral 2 1
After failure to get into #NSG, MODI JI should start
pushing to get a entry into #EU, Britain's vacant place awaits
for India. #Brexit
neutr
al Brexit neut
ral 2 2
The idea that UKIP will now disband, its mission
accomplished, is delusional. What emboldened reactionary
ever gave up their fight? #Brexit
negat
ive Brexit nega
tive 2 2
“I still have my ice cream. How can #Brexit be a big
deal.” - @yogurtearl, while eating ice cream.
positi
ve Brexit
neut
ral x 2 2
Sacramento: Tune into @kcranews at 5PM to catch our
very own Theo Slater provide our campaign's reaction to
#Brexit #BrexitVote.
neutr
al Brexit neut
ral 1 2
Page 64 of 76
http://Brexit101.com is for sale
http://ht.ly/LS98301BKIq #Brexit #BrexitVote #UK #Trump
#Cameron #Britain #EU #Greece #FX #Zika #Grexit
neutr
al other neut
ral 2 2
#Brexit Get out of the stockmarket now! Go see your
local ResiShare agent http://bit.ly/28IlT3x
negat
ive economy
nega
tive 2 2
ICYMI- Dow plunges 611 Points, British PM David
Cameron to resign, England to free from EU... #brexit lots
going on http://fb.me/7RgbzHxU0
neutr
al Brexit neut
ral 2 2
Amazing, Blair involved in illegal war killing 100s of
thousands no 1 tried 2 remove him,Corbyn in job a wet week,
coup against him #brexit
negat
ive
celebrities/p
oliticians nega
tive x 2 2
So does this mean we get a football team in England?
#Brexit #Nfl
neutr
al other
posit
ive 3 3
I just read that France is planning to send the thousands of
refugees in Calais towards UK. #brexit #immigration
neutr
al Brexit
neut
ral 2 2
What the hell did you do uk? #brexit #banksy
#yearofthemonkey #day137 #stupid #uk #eu #exit…
https://www.instagram.com/p/BHDnfFsj8DM/
negat
ive Brexit nega
tive 2 2
#Brexit is the warning to the EU leadership after Greek
referendum. They did not get it last year I hope they do now!
@EU_Commission
negat
ive other posit
ive 2 2
The frustrated Cold War warriors seeing #Brexit through
the prism of their paranoia about Vladimir Putin appear to be
revving their engines.
negat
ive Brexit nega
tive 2 2
@ABCNews24 LOL give me a 'Dog's Brexit' anyday over
an AbbottTurnbull government mate! #brexit was Rich v's
Poor, Far Right v's Centre Left
negat
ive Brexit nega
tive 1 1
Local Leave supporters 'pleased' & 'happy' with #Brexit
referendum result: http://chattelevision.ca/__news/local-leave-
supporters-pleased-and-happy-with-eu-referendum-result/ …
positi
ve Brexit posit
ive 3 3
Oh no #Brexit , how could you do this to large investors!
Today was mildly annoying, caused by "uncertainty".
#traderphobic #EUref #UK
negat
ive economy nega
tive 2 2
Now keep the promise of £350m a week for our #NHS -
Sign the petition: #EuRef #Leave #Brexit
https://you.38degrees.org.uk/petitions/invest-ps350-million-
saved-from-eu-in-nhs-by-2018?bucket=fb&source=twitter-
share-button … via @38_degrees
neutr
al Brexit
posit
ive 3 1
#Brexit sounds like a breakfast cereal #comedy
https://vine.co/v/5u7ntpa5B3U
neutr
al Brexit
neut
ral 3 2
MUST-READ #Brexit commentary: "But first, we will
have to think, probably more deeply than ever."
neutr
al Brexit
neut
ral 2 2
I wanna find my fellow Jubilee line riders who took down
the drunk Welsh #brexit supporter.
neutr
al other
neut
ral 2 2
Crude oil prices slammed after Britain votes to leave EU
http://klou.tt/104hlokjipjzm #brexit #oilprice #oilandgas
#petroleum
negat
ive economy nega
tive 3 2
Wonderfully succinct statement #Brexit #wtfbritain positi
ve Brexit
nega
tive x 2 2
You know, seriously, it's vapid & arrogant comments
from talking blow-dry heads like this that CAUSED #brexit. :/
negat
ive Brexit
nega
tive 2 2
#brexit today's removal of Paul Day's St Pancras "Meeting
Place" lovers statue. Things are happening fast.
neutr
al other
neut
ral 2 3
Watching Underworld at #Glastonbury2016 tonight in the
aftermath of #Brexit, I miss the 90's more than ever. Such
innocent times. Such hope.
negat
ive Brexit nega
tive 2 2
The club regret to confirm that the transfer of
@Ibra_official has collapsed due to the fall in value of the
pound as a result of #Brexit.
negat
ive other nega
tive 2 2
We owe so much to Nigel Farage for his unwavering
commitment to the #Brexit cause. What an historic day for
Britain #IndependenceDay
positi
ve Brexit posit
ive
2 2
Saying "everyone" online talking about #Brexit is as
smart as their dog inherently is discrediting those who are
talking
negat
ive other nega
tive x 2 2
Page 65 of 76
@realDonaldTrump so busy promoting his biz, his initial
reaction to #Brexit was, "there's nothing to talk about."
negat
ive Trump
nega
tive 2 2
Honestly most elites seem to think everyone needs a
daddy mommy bureaucrat to manage their lives for them. More
effete than elite. #Brexit
negat
ive Brexit nega
tive 2 2
New pictures of referendum result emerge... #brexit
@Oldfirmfacts1
neutr
al Brexit
neut
ral 2 2
This prerequisite of being a sociopath to have power is
getting a bit tedious now. #Brexit
negat
ive Brexit
nega
tive 2 2
.@SketchesbyBoze Perhaps this whole mess could've
been avoided if #Brexit was named Brexity McBrextface
negat
ive Brexit
nega
tive x 3 3
END OF THE EU? #Germany warns FIVE more
countries could leave Europe after #Brexit | World | News |
Daily Empress
neutr
al Brexit neut
ral 3 3
I regret voting for #Brexit under these false pretenses. negat
ive Brexit
nega
tive 2 2
Britain will be better off just like a spun out Corp from a
conglomerate, give 'er a day or two ;) #Brexit
positi
ve Brexit
posit
ive 2 2
there's no perfect society but Britain voted for worst out
of two. However 48% of UK ppl ain't happy with it #Brexit
negat
ive Brexit
nega
tive 2 2
I heard this great line on the @marklevinshow "when
immigrants don't assimilate it's called colonisation."
#immigration #Brexit #vidcon2016
negat
ive Brexit nega
tive 2 2
So, appears I successfully avoided making premature, not
informed enough direct correlation tween #brexit & U.S. 2016.
Mission Accomplished
positi
ve other nega
tive 2 2
With revolution life is so much better #Brexit positi
ve Brexit
posit
ive 3 2
The UK is like a kid that's been threatening to run away
from home, finally do & then immediately regret it #Brexit
negat
ive Brexit
nega
tive 2 2
Yep, him too. Saw a woman on #newsnight crying with
joy because she thought #Brexit had saved the NHS.
positi
ve Brexit
nega
tive 2 2
A day full of sad news. #Brexit then @Yellowcard
announce their end. Last tour tickets go on sale on payday,
seems like fate. Have to go!
negat
ive other nega
tive
2 2
Hey older generations in the UK. Thanks for putting that
final nail in the coffin for the rest of us...You CUNTS.
#BrexitVote #Brexit
negat
ive Brexit nega
tive 1 3
The #majority imposed their will on a very significant
minority in the #Brexit referendum
negat
ive Brexit
nega
tive 2 2
holy shit Nigel Farage has an Alan Partridge voice you
should have picked up on this British people! #brexit
neutr
al
celebrities/p
oliticians
nega
tive x 2 2
@realDonaldTrump You had NO FUCKING IDEA what
#Brexit or #BrexitVote was 2 days ago. LMAO
duhhhhhhhhhhhhhhhhh
negat
ive Trump nega
tive 2 2
Well well well, an appropriate song for the #Brexit result!
Sex Pistols Anarchy in The UK
positi
ve Brexit
nega
tive x 1 2
Sorry to see this happening... #Brexit negat
ive Brexit
nega
tive 3 3
Is this #Brexit going to mess with my UPC/VirginMedia
because my internet has been shit all day
negat
ive other
nega
tive 3 2
Dominar Farage strikes again... #farscape #Brexit #EUref negat
ive Brexit
nega
tive 3 3
#Brexit #GOT #GameofThrones The 'Brexit' referendum
and 'Game of Thrones' aren't all that different
http://mashable.com/2016/06/23/brexit-game-of-
thrones/#MQSrpW_B805A … via @mashable
neutr
al other
nega
tive 2 2
Mark it down, folks who are against #Brexit & calling the
#Leave folks racist, bigots, & xenophobes don't have a clue
about the real issues
negat
ive Brexit posit
ive 2 2
Buy Pounds .... Save Euros ... #Brexit neutr
al economy
neut
ral 3 2
By George, he gets it! Britain voted for #Brexit because it
wants to be #Canada http://pllqt.it/Tn6552
positi
ve Brexit
nega
tive x 2 2
Page 66 of 76
IMO David Cameron & Jeremy Corbyn should've put
politics & campaigned together on the platform of staying in
the EU... #Brexit #BrexitVote
negat
ive Brexit nega
tive
2 2
Dow jones took a MASSIVE blow today due to #Brexit.
This is a sneak peak at what would happen under Trump.
negat
ive economy
nega
tive
3 2
Victoria Nuland, Assistant Secretary of State, supporting
#Brexit in 2014, before it was mainstream.
neutr
al Brexit
neut
ral 2 2
#Brexit has led to some great memes #BrexitVote
#politics
positi
ve Brexit
neut
ral 3 3
This Dude Who Thought His #Brexit Vote Wouldn't
Matter Is A Valuable Lesson For All Of Us
https://www.mhb.io/e/1c6ei/cp
negat
ive other nega
tive 2 2
#Brexit is really complicated. I think the UK will be better
off in the long run, but I'm not certain.
positi
ve Brexit
posit
ive 2 2
Hillary picked to be on the wrong side of history for the
951st time. @realDonaldTrump #Brexit
negat
ive
celebrities/p
oliticians
nega
tive 2 3
Right, off to bed. Pretty tired so it'll be nice to lay my
head down on my soon to be de-regulated pillow
@iamjohnoliver #Brexit
positi
ve other posit
ive 1 2
Going in on @bondibeachradio in a couple of minutes
with #brexit bangers. Some geezer classics, some euro power.
neutr
al other
neut
ral 3 2
Did #vapers sway the #brexit vote? Perhaps everyone has
to deal with #Article50 because of #Artilce20 of the #TPD
negat
ive Brexit
nega
tive 1 2
#Brexit example of democracy in action. You must suck
and shut up
positi
ve Brexit
posit
ive 2 2
#Brexit today, #Libexit in a week. #auspol #ausvotes neutr
al other
neut
ral 3 3
From Rule Britannia to Cool Britannia to Fool Britannia
#brexit
negat
ive Brexit
nega
tive 1 1
Sigh. You had one job; #Britain #brexit negat
ive Brexit
nega
tive
2 2
Surprised #Sutton was one of few #London boroughs to
vote #Leave as was a @LibDems seat until 2015 #Brexit
@scullyp
negat
ive Brexit neut
ral
1 2
The #EU is far more about the New World Order of One
World Government than it is about a Global economy. #NWO
#Brexit @morningmika @brithume
negat
ive EU
posit
ive 2 2
#Brexit #history - future history exam question(s) neutr
al other
neut
ral 2 2
Super serious situation, but I chuckled. #Brexit Peace to
all OUR UK family! #Repost @tcb… http://itsOURshow.net
positi
ve Brexit
nega
tive 2 2
TheEconomist: Calls for a second Scottish independence
referendum grow louder after #Brexit …
neutr
al
Scottish
referendum
neut
ral 3 2
Before and after, my grass lost the referendum too
#grassxit #brexit , see what's left?
negat
ive Brexit
nega
tive 2 2
Damn you #Brexit supporters. I lost almost $10K in my
retirement today because of you and I'm from the U.S. I hope
you feel a similar pain!
negat
ive Brexit nega
tive 2 2
Does spellcheck have any part in Western Civilization?
#Brexit
neutr
al other
neut
ral 2 2
Nice work @CommBank leaving customers stranded
without access to their OWN money in UK cos it costs YOU
$$ #Brexit
negat
ive other nega
tive 1 1
Age old struggle between freedom vs. security. When
security just wasn't all that secure the people chose freedom.
#Brexit
positi
ve Brexit posit
ive 2 2
#Brexit generational gap is unbelievable -
http://www.cbc.ca/1.3650826
negat
ive Brexit
neut
ral 2 2
#Brexit: The Consequences Of Xenophobic Nationalism
https://theobamadiary.com/2016/06/24/brexit-the-
consequences-of-xenophobic-nationalism/ … via
@TheObamaDiary
negat
ive Brexit
nega
tive 2 3
Right behind you Marsha and Robert!!
#MakeAmericaGreatAgain #Brexit
positi
ve USA
posit
ive
3 1
Page 67 of 76
Haven't seen much excitement over the #Brexit vote.
Mostly a lot of despair and anger. I don't know if I should feel
sorry for the UK.
negat
ive Brexit neut
ral 2 2
@EricIdle #Brexit voters reminds me of the Crimson
Permanent Assurance pirates. Have they interrupted our lives
and ruined the world?
negat
ive Brexit nega
tive 2 2
END OF THE EU? Germany warns FIVE more countries
could leave Europe after #Brexit
neutr
al other
neut
ral 3 3
The #brexit discussion on Hardball dismissing Bernie and
both D & R voters for not embracing the "way we do things".
Still don't get it!
negat
ive USA neut
ral 2 2
Education matters, sure, but it's also a privilege. Scary
thing is that it's been becoming more and more of one. So ...
your move? #brexit
negat
ive other
nega
tive 2 2
How do you two on 'team globalist' feel about being on
the wrong side of history today? #Brexit #IndependenceDay
negat
ive Brexit
neut
ral 2 1
#Brexit Britain's biggest mistake since the 1773 Tea Act negat
ive Brexit
nega
tive
2 2
I take this as Target forewarning us of the disastrous
#Brexit fallout. Just the title, not the hackneyed plot.
negat
ive Brexit
nega
tive
2 2
Look out for Ed Sheerans Brexit cash in single "You need
me, I don't need EU" #edsheeran #Brexit #BrexitVote
#EUreferendum #EUref #jokes
neutr
al Brexit neut
ral x 2 2
Up until this weekend I thought #Brexit was some sort of
cracker for tea.
neutr
al Brexit
neut
ral x 2 2
don't even use #Brexit you were for Remain your
continual lying will not get you anything traitor!
negat
ive Brexit
posit
ive 2 2
Jermaine Pennant asks the most important question of the
day http://dailym.ai/28RBRZW via @MailOnline #brexit
#hesaidwhat?
neutr
al other neut
ral 2 2
If the EU savings aren't going to the NHS as promised,
where pray tell are they going? #Brexit
neutr
al Brexit
nega
tive 2 2
#IVotedLeave #England #brexit I'm not British but i hope
the best for England
positi
ve Brexit
posit
ive
2 2
Being united w other countries is something so important
- it provides a sense of safety & solidarity. I am so, so sorry
Britain. #Brexit
negat
ive Brexit nega
tive
2 2
Of all UK tweets responding to #Trump stupidity, this is
my fave: "Scotland hates both #Brexit and you, you mangled,
apricot, hellbeast."
positi
ve Trump nega
tive
2 2
i miss original recipe steven colbert. hed be killing #brexit
watching him on the late show feels like watching jordan play
baseball
negat
ive other nega
tive 2 2
Stay tuned for some amazing Crop Circles. #brexit neutr
al other
neut
ral 1 3
Disaster Ahead! Trump on Brexit: America is next
http://cnn.it/28VPZjy #Brexit #Election2016
negat
ive Trump
neut
ral 3 2
#Brexit Won! What does it mean to #domain investors? neutr
al economy
neut
ral 2 2
Or is it the fear of Changing the Paradigm or is it the Fear
that the Paradigm will NEVER change? #BrexitVote
#BREAKING #Brexit
neutr
al Brexit neut
ral 2 3
EU can’t go on forever without earning consent of the
governed: @InklessPW http://on.thestar.com/28WI4Cr #euref
#Brexit #democracy #consent
negat
ive Brexit posit
ive 2 2
Hats off to the German Foreign Office! #BrexitVote
#Brexit #EURef
positi
ve other
neut
ral 3 2
#Brexit & #Czechout & #Finish, oh my! Denmark too. Is
#EUXit next? #MAGA #Trump2016
neutr
al Brexit
neut
ral x 3 2
#Brexit #DavidCameron I wonder will Cameron be see
as. #nevillechamberlain figure "Europe in our time" figure.
Disaster #PM
negat
ive Brexit nega
tive 2 2
Just kidding One of the key reasons you have to be careful
when trusting populist propaganda. #brexit
negat
ive Brexit
nega
tive 2 2
Page 68 of 76
LOL. I heard someone say, "make England great again."
That sounds awfully familiar... #Brexit
#MakeAmericaGreatAgain
negat
ive USA nega
tive 3 3
1/ A possible historical parallel to London post #brexit is
Montreal
neutr
al Brexit
neut
ral 2 2
At what point does #Leave #Brexit get called out for flat
out lying in campaign commercials? Nope. NHS won't get that
money. #UKreferendum
negat
ive Brexit nega
tive 3 2
British Lose Right to Claim That Americans Are Dumber
http://www.newyorker.com/humor/borowitz-report/british-
lose-right-to-claim-that-americans-are-dumber … via
@BorowitzReport #Brexit
neutr
al Brexit
nega
tive x 3 2
"#Brexit is not gonna play well in Washington."
#BBCWorld
negat
ive Brexit
nega
tive 2 2
Should we start calling this what it is - the rise of fascism
in #Britain #brexit #EURefResults
negat
ive Brexit
nega
tive 2 2
After #Brexit? Departugal. Italeave. Fruckoff. Czechout.
Oustria. Finish. Slovlong. Latervia. Byegium. = Germlonely
neutr
al Brexit
neut
ral x
3 3
Separatists in Scotland and terrorists in Northern Ireland
are cynically calling for new referenda to achieve their raisons
d'être. #Brexit
negat
ive other neut
ral
2 2
Don't like bureaucracy but a lot of #Brexit/Trump votes
are based on fear and hate. That's what I'm against more than
anything. Depressing.
negat
ive Trump nega
tive 2 2
Just something to lighten the mood a little bit.
@naomivowles you can never go wrong w/ Spice Girls #Brexit
http://www.youtube.com/watch?v=SoxxHeBJmz8&sns=tw …
positi
ve other nega
tive 3 2
BostonGlobe: After #Brexit, European property investors
may see better value — and stability — in Mass. …
positi
ve economy
neut
ral 3 2
#Brexit would have been defeated if #EU had accepted
NO to #treaties,listened to ppl & reformed.predict
#dominoeffect
negat
ive EU nega
tive 2 2
People "going with thier gut" instead of their brain has
never lead to anything other than rasict dumbassery. #Brexit
negat
ive Brexit
nega
tive 2 2
I wanted #Brexit hugely. positi
ve Brexit
posit
ive 3 3
@MSNBC @amjoyshow Lindsey Lohan tweeting about
#Brexit is no more “Breaking News” than me tweeting about
it!
negat
ive celebrities/p
oliticians
neut
ral 2 3
@realDonaldTrump Florida is in Scotland? Or is Scotland
in Florida? I'm so confused. #brexit #yousirareanidiot
negat
ive Trump
nega
tive 2 2
Sweet, I just found £8 down the back of the sofa. Thanks
#Brexit
positi
ve other
neut
ral x 2 2
How can we criticize the #brexit when our president flies
on a plane named after a shoe
negat
ive Trump
nega
tive x 2 2
Serious question: does anyone have a time machine?
#Brexit
negat
ive other
nega
tive x 2 3
BostonGlobe: #Brexit was spawned by tensions over
globalization, President Barack Obama says …
negat
ive Brexit
nega
tive 2 2
#Brexit is all shits & giggles to me until I check my BT
European funds
negat
ive Brexit
nega
tive 3 2
lol. Looking forward to the day the english come crawling
back. #Brexit #Leave #brexitfail
positi
ve Brexit
nega
tive x 3 2
Almost incredible that none of the #Eurocrats seem to
have had a contingency plan for #brexit, but should have
expected it.
negat
ive Brexit nega
tive 2 2
I use to think I was proud to be British but not proud of
Britain, but now, im not so sure about the second part #Brexit
negat
ive Brexit
nega
tive 2 1
if someone had said that today’s economic situation
would be the immediate outcome of #brexit how many would
have believed? (vs more FUD)
negat
ive economy nega
tive x 2 2
Out of the #Brexit ashes I feel sorry for the Japanese. The
Japan economy desperately needs a lower Yen but the Brexit
computer says "No"
negat
ive economy
neut
ral x 2 2
Page 69 of 76
This is a pretty perfect response. @dominicnahr #brexit
#montypython
https://www.instagram.com/p/BHDnVGoA3Ve/
positi
ve other neut
ral 3 2
Finally logged off from work. Thanks for the overtime
cash #Brexit
positi
ve other
neut
ral 2 2
There is some satire in this #brexit negat
ive other
nega
tive 3 3
Rise of #Bitcoin and #Gold as #Brexit Turns into Reality neutr
al economy
neut
ral 2 2
Israel and America now should leave the UN #Brexit neutr
al other
posit
ive 3 2
It can't be "take back control" for England and "lose all
control" for Scotland & NI. #UnitedIreland #Indy2 #IndyScot
#Brexit
negat
ive Brexit nega
tive 2 2
considering the breakdown of voting in #Brexit I propose
a new country form and its name should be Scotirelondon
#scotirelondon
neutr
al Brexit nega
tive x 2 2
Have the four horseman appeared yet? #Brexit negat
ive Brexit
nega
tive x 2 2
When we leave EU we won't be protected by the Privacy
Shield agreement with the US. Goodbye privacy, thanks
#Brexit.
negat
ive Brexit nega
tive 1 2
I think if #Texit happens it will start a domino effect just
like #Brexit did
neutr
al other
nega
tive 3 2
The only good thing to come out of the #Brexit is the
dearth of insults being hurled at @realDonaldTrump by the
lovely people of Scotland
positi
ve Trump
nega
tive x 2 2
#Canada pls keep it together so we have somewhere to go
if Drumpf becomes POTUS #Brexit :(
negat
ive Trump
nega
tive 3 2
Come to think of it, #Brexit does have a smoky taste to it negat
ive Brexit
nega
tive 2 2
$2.7 TRILLION lost on global markets after
@RupertMurdoch has his #Brexit dreams come true. People
will die from impacts of such losses.
negat
ive economy
nega
tive 2 2
Stop being afraid of what could go wrong and start being
excited about what could go right . #Brexit #VotedLeave
positi
ve Brexit
posit
ive 3 2
"Democracy is the theory that the common people know
what they want, and deserve to get it good and hard." -H.L.
Mencken #Brexit #BrexitVote
neutr
al other
posit
ive 2 2
Thought of #TREXIT tantalizing, not happening. After
#Brexit the Scots hate him, where shall The Donald go?
Instead #ImWithHer
negat
ive Trump
nega
tive 2 2
Despite the alerts, did not open Robinhood once today.
#Brexit #dontpanic #letthatpoundcomedown #ineverbeentogb
positi
ve other
nega
tive 2 2
..And the Oscar as worst world leader ever goes to..
#Cameron #Brexit #ByeByeUKEP #UKreferendum
negat
ive
celebrities/p
oliticians
nega
tive 2 2
U2 said it best "And I wait without EU / With or without
EU" #Brexit
neutr
al Brexit
neut
ral x 3 2
My point is the racists that have always been there now
seem to think it is acceptable to be openly racist since #Brexit
negat
ive other
nega
tive 2 2
Sloooow down #MariaBartiromo...it's not the end of the
world yet. #Brexit #greta
positi
ve Brexit
posit
ive 2 2
#Brexit' to be followed by Grexit. Departugal. Italeave.
Fruckoff. Czechout. Oustria. Finish. Slovakout. Latervia.
Byegium.
neutr
al Brexit
neut
ral x 3 3
isn't that exactly what voters HATE. Pos afraid for
THEIR political futures..the country's be damned, be it #brexit
or NRA
negat
ive Brexit
nega
tive 2 2
The first task of the new #ToryLeadership will be how to
diminish #UKIP #bestofenemies #Brexit
neutr
al Brexit
nega
tive 3 3
I bet right now almost everybody in Florida is looking at
this whole #Brexit thing and thinking, "Aw, man, did I just shit
my pants again?"
negat
ive Brexit
nega
tive x 2 2
Every time someone says #brexit our cats lift up head
with big eyes as if it means #biscuit ... at this rate they'll never
learn English.
neutr
al other
neut
ral x 3 3
Page 70 of 76
#Brexit was just a massive hoax so we could have a
global meme competition.
negat
ive Brexit
nega
tive x 2 2
Does this mean no more Tim Tams? #Referendumb
#Brexit
negat
ive other
nega
tive x 3 3
Brits were tired of hearing from "outside experts" #brexit
#lohanknew
negat
ive
celebrities/p
oliticians
nega
tive x 2 2
I bet that's not the first time Obama's been mentioned in
the same sentence with an "eight ball." #Brexit
neutr
al
celebrities/p
oliticians
neut
ral x 2 2
I know it is not good for me, but, on days when Britain
chooses to #Brexit, I like to drink a Coke and eat a cookie.
#EURefResults
negat
ive other
nega
tive 2 2
On some level UK after #Brexit must feel a little like US
after GW Bush reelection. Mainly in how the rest of the world
is like "Seriously?"
negat
ive Brexit
nega
tive x 2 2
No one saw #Brexit coming. No one will see FBI
indictment recommendations of Hillary coming. 2016 is a wild
year. Be ready.
negat
ive USA
nega
tive 2 2
The average life experience of those whining about
#Brexit seems to be about 12 years.
negat
ive Brexit
posit
ive x 2 2
#OutMeansOut - Let's leave this mess before it all
collapses all around us. #LeaveEU #Brexit
negat
ive Brexit
posit
ive 3 3
As someone who works closely with highly skilled
migrants, I can say I value your work and you are an asset to
this country #brexit
positi
ve other
nega
tive 3 2
Did that 1% growth from the $50billion business tax cut
after 10 years include modelling in a #brexit scenario?
#ausvotes
neutr
al economy
neut
ral x 2 2
Sebastian was right, can I become a mermaid now pls
#Brexit #EURefResults
negat
ive Brexit
nega
tive x 3 2
If some of you people had your way, the USA would've
never existed. Would've been happy paying taxes to the British
for a lifetime. #Brexit
positi
ve Brexit
nega
tive 2 2
#HiddleSwift or #Brexit, don't make me choose!
(Meanwhile, in America...) #TGIF and the markets are closed,
whew!
neutr
al other
nega
tive x 2 2
Don't brex my heart, Never leave me again #Brexit
#BrexitVote #ToniBraxton #Nailedit #Youaresonotfunny
neutr
al Brexit
neut
ral x 2 2
When you see that Trump endorsed UK leaving EU, you
can realise how stupid that idea it is. #Brexit
negat
ive Trump
nega
tive 2 2
its begun, The Begining of the END #Brexit #Damn negat
ive Brexit
nega
tive 2 2
Now keep the promise of £350m a week for our #NHS -
Sign the petition: #EuRef #Leave #Brexit
https://you.38degrees.org.uk/petitions/invest-ps350-million-
saved-from-eu-in-nhs-by-2018?bucket=blast&source=twitter-
share-button … via @38_degrees
neutr
al
Brexit
posit
ive 3 1
#Mckinney #FreddieGray #GeneleLaird #Brexit
#immigration The news has been exhausting these past few
days
negat
ive other
nega
tive 2 2
All I have to say about #Brexit #love #ignorance #hate
#nofear #peace #tolerance…
https://www.instagram.com/p/BHDnRjwA-_k/
neutr
al Brexit
nega
tive 1 3
So Black & brown ppl should stay in their "native"
countries? I could be ok w/ that, white ppl, if you also stay in
"yours." #Brexit #Trump
negat
ive Trump
nega
tive 2 2
Churchill must be smiling down on #obama as Briton
follows her history & refuses to surrender, once again #brexit
#tcot #p2 #gop #dem
positi
ve Brexit
posit
ive 2 2
#Cameron & #Osborne tried to call the bluff of Tory
troublemakers for party reasons -it's backfired badly #brexit
negat
ive Brexit
nega
tive 2 2
Brexit big blow to UK science, say top British scientists-
https://www.theguardian.com/science/2016/jun/24/brexit-big-
blow-to-uk-science-say-top-british-scientists?CMP=twt_a-
science_b-gdnscience … #Brexit #Science
negat
ive
Brexit
nega
tive 2 2
This outcome was set from the stary, calling it #Brexit
instead of Bremain.
neutr
al Brexit
nega
tive 2 2
Page 71 of 76
#Brexit affirms what's innate about bureaucracy - in spite
of intentions it grows, empowers elites, squelches public and
self determination
negat
ive Brexit
nega
tive 3 3
I guess all those UK Citizens of every color voting for
#Brexit were racists too?
negat
ive other
nega
tive x 2 2
"I'm divided." British expat in Co talks about uncertainty
in the UK after #Brexit. Her story @6.
negat
ive Brexit
neut
ral 2 2
#BREXIT good timing!!! EU nazi caliphate waiting just
on other side of the channel. @FoxNews
positi
ve Brexit
posit
ive 3 2
David #Cameron, you are so outta here! #Brexit #EU
#EUreferendum #notmyvote #Johnson
negat
ive Brexit
nega
tive x 2 2
We may end up w/ @BorisJohnson and
@realDonaldTrump WOW!!! FDR and Churchill r checking
w/ God about reincarnation #Brexit #insanity
negat
ive celebrities/p
oliticians
nega
tive x 2 3
Relatedly I’m going to compile a reading list for myself
on possible EU reforms bc clearly that was a huge
(understandable) #Brexit factor.
neutr
al other
neut
ral 2 2
So....... the Queen kills everybody now, right? #Brexit neutr
al Brexit
nega
tive x 2 2
#Brexit well wishes from the neighbours! positi
ve Brexit
posit
ive 2 2
BRB. Reading all about #Brexit and possibly planning a
trip to England ASAP before things get too crazy.
neutr
al other
neut
ral 3 3
I bet a lot of UK businesses are feeling pretty bummed
about their .eu domain extensions... #brexit #EURefResults
#EUref #firstworldproblems
negat
ive other
nega
tive x 2 2
This bozo found a legal loophole... #Brexit but with the
"benefit" of unfettered immigration into the UK...
negat
ive Brexit
nega
tive 3 2
Nice to see how it takes #brexit to make @guardiannews
see the private sector as more than just an iniquitous force that
needs to be taxed
negat
ive Brexit
nega
tive 2 2
Looks like the Coudenhove-Kalergi plan and his book
Praktischer Idealismus for the European Union are failing
thanks to Britain. #Brexit
negat
ive other
nega
tive 2 2
Whatif?@bernardchickey #Brexit #RemainINEU UK had
heard non political/self-interest view, just facts - same result?
neutr
al Brexit
neut
ral x 2 2
Day 1 of #brexit Still in the EU. neutr
al Brexit
neut
ral x 3 2
Following that course, disintegration of social bonds and
solidarity is the result. That's where we are where we are.
#Brexit
negat
ive Brexit
nega
tive 2 2
Will England remain at Epcot Center? #Brexit neutr
al Brexit
neut
ral 2 2
I am starting to learn #French and using more of my
#german ... #Brexit #tragic
negat
ive other
nega
tive 3 3
@David_Cameron 's pull out game is really weak. You're
supposed to pullout BEFORE you mess up "her" life. RIP
Britain. #BrexitVote #Brexit
negat
ive Brexit
nega
tive 2 2
So when do we start sending ration packs to the UK.
#Brexit
neutr
al Brexit
nega
tive x 1 1
Soooo what if Trump bought Texas and renamed it
Trumpxas. Then we can vote for a #Trexit... #Brexit
neutr
al Trump
nega
tive x 2 2
I just woke up and there is a severed horse head next to
me... What does this mean? Did I piss off the Mafia? #help
#Brexit #Mafia #horse
negat
ive other
nega
tive x 2 2
Two #Brexit fears are hard to reconcile 1. Scotland, N
Ireland leaving the UK to join EU while 2. EU countries like
France move toward exit.
negat
ive Brexit
nega
tive 1 2
His cluelessness of #Brexit neg impact on Sctland as he
talked is n xmple of infantile diplomacy.
negat
ive Trump
nega
tive 2 2
What can the EU do that it never did before against ISIS,
#Brexit changed nothing UK doing all heavy lifting and paying
EU invites them in
negat
ive EU
posit
ive 2 2
I just hope my 401(k) didn't shit the bed today over
#Brexit. It probably did tho.
negat
ive other
nega
tive 2 2
Page 72 of 76
“Anyone else think #brexit sounds like a super healthy
digestive type of biscuit? 'Ooeck owsabout you pass me the...
http://fb.me/5jalEFbdB
neutr
al Brexit
nega
tive x 2 2
France is now the 5th largest economy in the world, took
just 5hrs, thanks to #Brexit
positi
ve economy
nega
tive x 1 2
I want #AMERICA back!! #MakeAmericaGreatAgain
#MakeAmericaSafeAgain #Brexit
neutr
al USA
posit
ive 1 1
.@lsarsour #Brexit is, or should be, a first step to local
autonomy and bio-regional governance, without which
#sustainability is impossible
positi
ve Brexit
posit
ive 3 3
Seems like my boys spent most of their time at HSS for
the last week or so. Wall Street was like being at a wake today.
#Brexit
negat
ive economy
neut
ral x 2 2
If Trump and the French national front celebrate the brexit
vote.... You know you've fucked up royally.... #Brexit
negat
ive
celebrities/p
oliticians
nega
tive 2 2
The latest The Makam News!
http://paper.li/igbariam/1347306962?edition_id=3e09c910-
3a67-11e6-b556-0cc47a0d15fd … #nbadraft #brexit
neutr
al other
neut
ral 3 3
We'll find out the effect of #Brexit on the performance of
British national teams very soon. #EURO2016, #England,
#Wales, #Ireland, #NIR
neutr
al other
neut
ral 3 2
Why are people encouraging the dismantling of the EU?
#Brexit
negat
ive Brexit
nega
tive 2 2
#greta who would you rather making the deals with
#Brexit @HillaryClinton or @realDonaldTrump this kind of
skews the election
neutr
al USA
neut
ral 2 2
If no fed rate hike now, mtg. rates should stay around
historic lows for a while. How long though and is that really
beneficial? #Brexit
negat
ive Brexit
nega
tive x 3 2
Just watched Jim Cramer on this morning's
@TheTodayShow telling an astonished Matt Lauer that the
economic impact of #Brexit is not so bad.
neutr
al economy
nega
tive 2 2
Americans have no respect for Obama why would the
Britts #Brexit #greta #Trump2016
negat
ive
celebrities/p
oliticians
nega
tive 2 2
Just for clarification since I've gotten 10+ messages on
this: No I don't think #Brexit was solely due on racism, clearly.
negat
ive Brexit
nega
tive 2 2
Colonises half the World and complains about
#immigrants. #Brexit
negat
ive other
nega
tive x 3 2
Seriously, this is so retarded my head hurts. #brexit got
the most votes on a referendum. You lost. Cry elsewhere
negat
ive Brexit
posit
ive 2 2
Ahh @JoyAnnReid brings up the “Anti Expert” element
of #Brexit. I look around & see debate against science &
experts in my own state often
negat
ive Brexit
nega
tive 2 2
The fascist, one-world tyrant-lovers cuss a lot when they
venture outside of their safe place to complain about #texit or
#brexit. Haha
negat
ive Brexit
posit
ive 2 2
We should have right to retain our EU citizenship even
after #Brexit.
neutr
al Brexit
nega
tive 2 2
Brits who voted to leave in #Brexit, to be safer, apparently
never heard of #DivideAndConquer
negat
ive Brexit
nega
tive 2 2
#Brexit I'm American, but I am also (by decision) Greek,
French, Italian, Spanish, Dutch, and Belgian. But I will never
be an #Englishwoman.
negat
ive other
neut
ral 1 2
American Media narrative bout #Brexit focuses on
THEIR stupidity while reportin Lohan tweet, views of
presumptive nominee Trump #GlassHouses
negat
ive USA
nega
tive 2 2
Utility stocks, along with Treasury bonds, serve as safe
haven during tumultuous times. @SmithRebecca
http://ow.ly/ldWK301CoUC #brexit
neutr
al economy
neut
ral 3 3
This is Great Britain not a 3rd world country. The elite
selling today will be buying next week, part of their scare
tactics. #Brexit
negat
ive economy
nega
tive 2 2
Can we solve #Brexit issues using the old "reset" method?
Switch off, leave Europe for 10 secs then plug ourselves bck
in? Oh wait... oops?
neutr
al Brexit
nega
tive x 3 3
Page 73 of 76
Shot Across #liberal Bow: Brits decided to ‘Make
England Great Again' →NEXT-UP #MakeAmericaGreatAgain
#Brexit #NEVERhillary @LouDobbs @greta
neutr
al USA
nega
tive 2 3
Hoping for Steve Harvey to come out and say the
#EUreferendum result was wrong lol. #Brexit #EURefResults
#NotMyVote
neutr
al Brexit
nega
tive x 2 2
Am I reading that right? Is the fact that Lindsay Lohan
tweeted about #brexit "Breaking News." Or did I have a
stroke? #msnbc
negat
ive celebrities/p
oliticians
neut
ral 2 3
Salute to British state and people to sticking to will of
majority and democracy, come what may #Brexit #Uk
negat
ive Brexit
nega
tive 2 2
I gotta be honest, it's bit of a relief to get confirmation that
America isn't the only country with stupid citizens. #Brexit
positi
ve USA
nega
tive 2 2
Last in first Out! #brexit neutr
al Brexit
neut
ral 3 3
Hey, United Kingdom, Imma let u finish, but USA had
the greatest #Brexit of ALL TIME!
positi
ve USA
neut
ral 1 3
oh boy, #texas is inspired to secede again based on
#Brexit. Not only is it illegal, but the real question is who
really cares for Texas?
negat
ive other
nega
tive 2 2
#NeverTrump goes to Scotland and talks about
sprinkler.Missed the whole point! #Brexit was crucial and
warranted some explanation from trump
negat
ive Trump
nega
tive 2 2
Thanks British #brexit twats, I'm feeling poorer today negat
ive economy
nega
tive x 1 2
Asked my wife if she heard about #Brexit and she said no.
Started to explain and she doesn’t even know about the EU. I
quit.
negat
ive Brexit
neut
ral 3 3
#Brexit is crushing victory 4 ppl against the
establishment. Get used 2 it. #EngalndLiberation #LeaveWins
positi
ve Brexit
posit
ive 2 2
@StephenNolan @bbc5live this Gerry guy is the
embodiment of smug, arrogant BBC leftwing metropolitan
self-proclaimed elites. Happy 4 #Brexit
negat
ive Brexit
posit
ive 2 2
An electoral college system sounds like a good idea right
abut now dunnit? #Brexit
neutr
al other
nega
tive x 2 2
We have an extraordinarily high level of international
reserves 177 billion dollars #Brexit #Mexico #PressRelease
positi
ve economy
posit
ive 3 2
#Brexit is similar to union-busting but not a union of dock
workers, it's a union of bankers, technocrats, oligarchs, fascists,
etc.
negat
ive Brexit
nega
tive 2 2
Looks like the EU has about 1 GB more free space now.
#Brexit
neutr
al EU
neut
ral x 3 2
Let love win! Let the religion of peace engulf your once
proud nation. Celebrate their tolerance! #onpoli #Brexit
positi
ve Brexit
posit
ive 1 1
@Lizabs68 By time #Brexit is complete, a majority of
those who voted & are still alive will be #Remain supporters.
#BrexitMustBeStopped
negat
ive Brexit
nega
tive 2 2
Talk of a 2nd referendum if EU gives us a new deal which
will appease the people voting #Leave Hopefully that comes to
pass #EUref #Brexit
neutr
al Brexit
nega
tive 2 2
This is shocking. “What is the EU”? asks citizens of the
UK, AFTER #Brexit.
negat
ive EU
nega
tive 2 2
Help my 'Get the fuck out of England' fund with this
#EURefResults design.
http://www.redbubble.com/people/lunarblaze/works/22256458
-dont-blame-me-eu … #EUreferendum #BrexitVote #Brexit
neutr
al
other
nega
tive x 3 3
Bed after a long two days. Three events today with one
topic #Brexit. People are uncertain but rational about what
happens next.
neutr
al Brexit
nega
tive 2 2
Nothing can save this day, but I guess we will always
remember it! #brexit
negat
ive Brexit
nega
tive 2 2
Guess I won’t have to deal with anymore Brits on holidy
who never tip #cheerio #brexit
positi
ve other
neut
ral x 2 2
Hey. England. If you wanna be a wanker go ahead… But
how about cutting Scotland and Ireland loose before ya drag
them down too! #Brexit
negat
ive Brexit
nega
tive 2 2
Page 74 of 76
What would it be called if the US decided to do a #Brexit?
U Sexit?
neutr
al USA
neut
ral x 2 2
#HistoryNotes #Brexit illustrates economic point no
school teaches: Money is Imaginary – Economies are Ideas –
they are what you make them
neutr
al economy
nega
tive x 2 2
There’s a vacancy, we already have a foot in the door via
Eurovision Australia might ass well have a crack! Let’s join
the EU! #Brexit
positi
ve EU
nega
tive x 2 2
Increase Google searches in that region of the wold asking
“what is #Brexit?” and “what is the UK?”
neutr
al Brexit
neut
ral 2 2
I'll bet all these Hollywood stars so knowledgable about
#Brexit are the same ones who were primate experts just a few
weeks ago. #clueless
negat
ive celebrities/p
oliticians
nega
tive x 2 2
#Brexit Angela Merkel's country faces having to pay an
extra £2.44billion a year to the annual EU budget once Britain
has left.
negat
ive celebrities/p
oliticians
nega
tive 2 1
DID YOU HEAR? The UK #Brexit took their country
back. Time for s to take USA back! Donate to Trump
Campaign now at
positi
ve USA
posit
ive 2 2
Both #Brexit and #Trump are disasters. No thanks. I'll
vote for @HillaryClinton and other sane people.
negat
ive
celebrities/p
oliticians
nega
tive 2 2
Will #Brexit disintegrate the EU? The EU needs a new
democratic constitution or it will disintegrate! Join #DiEM25
community, save EU!
negat
ive EU
nega
tive 3 2
To understand #brexit, the immigration issue etc...is to
understand y they are leaving home in the first place. Who is
shaping these events?
neutr
al Brexit
neut
ral x 2 2
The British call their Independence #Brexit... maybe we
should call ours #Mexit Vote #Trump2016
neutr
al USA
posit
ive 2 1
One positive from #Brexit is that it has shown which areas
of the UK need better funding for education.
positi
ve Brexit
nega
tive 1 2
Congratulations on your #Brexit from the globalists, hope
to do the same in Nov #Trump2016 #MakeAmericaGreatAgain
positi
ve USA
posit
ive 1 1
The EU's 'techno party' is hollowing out democracy
http://openermedia.blogspot.com/2015/05/the-eus-techno-
party-is-hollowing-out.html … #Brexit LoiTravail David
Cameron Scotland
negat
ive
other
nega
tive 2 2
If Sarah "Dumbass" Palin is excited about your decision..
you just made a HUGE global boner of a mistake. #Brexit
negat
ive
celebrities/p
oliticians
nega
tive 2 2
My thoughts and prayers go out to all the people affected
by this #Brexit fiasco
neutr
al Brexit
nega
tive 2 2
I'd be more upset about #brexit if it didn't sound so much
like #breakfast . Instead, I'm just hungry.
neutr
al other
nega
tive x 2 3
So, that was the dress rehearsal. Now that you Leavers have
seen the effects of your vote, would you like to try that again?
#Brexit
negat
ive Brexit
nega
tive x 2 1
Above anything else, I genuinely feel sad for our country
today. #EUref #Brexit #notmyvote
negat
ive Brexit
nega
tive 2 2
I never thought I would live to see it broken up. #Brexit negat
ive Brexit
nega
tive 2 2
IMHO #Brexit fallout is totally overblown. People are
making money off the rampant panic & uncertainty. Folks need
to calm the fuck down.
negat
ive Brexit
nega
tive 2 2
So long….and thanks for all the bypasses. #Brexit neutr
al Brexit
nega
tive 1 2
Something is changing........ #EURefResults #Brexit neutr
al Brexit
neut
ral 2 2
It’s failing #Brexit negat
ive Brexit
nega
tive 3 2
Maybe he thought Brexit was the guy's name? Man Who
Voted For #Brexit Is 'A Bit Shocked' His Vote Counted,
'Worried'
negat
ive Brexit
nega
tive x 2 2
#Brexit is the logical result of Thatcher's 'Big Bang.'
State-sanctioned inequality + the creation of ultra-capital &
#financialization.
negat
ive Brexit
nega
tive 2 2
Page 75 of 76
Lotsa peeps loving my fiction on #Brexit -> #Trump ->
World War 3 via Scots, Texit, Huxit, Grexit, Bletchit, Putin
neutr
al other
nega
tive 2 2
#Brexit processing takes years.dont see why #US #Stock
to be affected now?instead it is a good hands to buy in!
positi
ve economy
neut
ral 2 2
#Brexit sounds like a planet in the #StarWars EU which
makes this a very confusing day.
neutr
al Brexit
nega
tive x 3 2
Don't know a lot about #Brexit, but i do enjoy watching
the Super Pundit Class hyperventilating and clutching their
pearls.
positi
ve economy
nega
tive x 2 2
Yes, stock markets will plummet. $$ interests only care
about making more $$ for the sake of itself. Common sense.
#brexit #financialization
neutr
al economy
nega
tive 2 2
I served in the UK at RAF Bentwaters/Woodbridge &
could not be prouder of them taking their country back!
#Brexit
positi
ve Brexit
posit
ive 2 2
Never underestimate the power of stupid people in large
groups! #Brexit #jokeofthecentury
negat
ive Brexit
nega
tive x 2 2
#UK left the European Union...I guess it's time to find
work elsewhere =_= Thanks #Brexit
https://youtu.be/I17j7vzFnN0 via @YouTube
negat
ive Brexit
nega
tive x 2 2
The next James Bond will just be him spending 2 hours in
passport control De Gaulle #Brexit #JamesBond
negat
ive other
nega
tive x 3 2
Will #brexit hurt English Football League? neutr
al other
neut
ral 3 3
I need the #Brexit jokes to stop until I can refill my
prescription
neutr
al other
nega
tive x 2 2
#BrexitAdam didn't think his vote would count? now you
know folks, your vote ALWAYS counts! #Brexit #BrexitVote
#BrexitOrNot #Election2016
negat
ive Brexit
nega
tive 2 2
Can we all just fast forward to 2017 instead? #Brexit
#DonaldTrump #RefugeeCrisis
negat
ive Brexit
nega
tive x 2 2
Thank you, @NicolaSturgeon, for demonstrating what
true, compassionate leadership looks like in the face of
adversity. #Brexit
positi
ve celebrities/p
oliticians
nega
tive 1 2
if #Brexit was Rigged Find out who Invested in Gold
Profited just like insider Trading Before 911 on thr Airline
Stocks SECURITY FIRM COMEX
neutr
al economy
neut
ral 3 2
#BREXIT – Americans! Watch & learn from a VOTE
based on revenge and xenophobia. If you vote unhinged, there
are consequences! #NeverTrump
negat
ive Brexit
nega
tive 2 2
#brexit is so catchy I love it positi
ve Brexit
posit
ive 2 2
Page 76 of 76