ESWC-17 Challenge: 4 rd Semantic Sentiment Analysis Portroz, Slovenia, May, 2017 CHALLENGE CHAIRS: Diego Reforgiato Recupero (University of Cagliari) Erik Cambria (Nanyang Technological University, Singapore) Emanuele Di Rosa (FINSA, Italy)
ChallengeIntroduction2017Portroz, Slovenia, May, 2017
1. INTRODUCTION 2. CHALLENGE TASKS 3. EVALUATION
[1] comScore/the Kelsey group. Online consumer-generated reviews
have significant impact on offline purchase behavior. Press
Release, November 2007. http://www.comscore.com/press/
release.asp?press=1928. [2] John A. Horrigan. Online shopping. Pew
Internet & American Life Project Report, 2008. [3] Lee Rainie
and John Horrigan. Election 2006 online. Pew Internet &
American Life Project Report, January 2007 [4]
http://www.umiacs.umd.edu/research/LCCD/
Goals
• Bridge research and industry at international level. This year
among the organizers: University of Cagliari (Italy), Nanyang
Technological University (Singapore), FINSA (Italy)
• Provide a forum for demonstrating the suitability of research
approaches in real-world context
• Share own experiences
Challenge Tasks
• Task #1: Polarity Detection • Task #2: Polarity Detection in
presence of metaphorical language
• Task #3: Aspect-Based Sentiment Analysis • Task #4: Semantic
Sentiment Retrieval • Task #5: Frame Entities Identification • Task
#6: Subjectivity and Objectivity detection • Most innovative
approach
Challenge Tasks
• Task #1: Polarity Detection • Task #2: Polarity Detection in
presence of metaphorical language
• Task #3: Aspect-Based Sentiment Analysis • Task #4: Semantic
Sentiment Retrieval • Task #5: Frame Entities Identification • Task
#6: Subjectivity and Objectivity detection • Most innovative
approach
Task #1: Polarity Detection The main goal of the task is polarity
detection. E.g. “Today I went to the mall and bought some desserts
and a lot of very nice Christmas gifts”, which should be classified
as positive.
Dataset[5] Train: 1M Amazon review, 20 domains, pos/neg balance
Test: 33,361 Reviews randomly selected across the same 20
domains
[5] Mauro Dragoni, Andrea Tettamanzi and Célia da Costa Pereira -
DRANZIERA: An Evaluation Protocol For Multi-Domain Opinion
Mining
Task #2: Polarity Detection in presence of metaphorical
language
The main goal of the task is polarity (positive, negative, neutral)
detection. E.g. “I just love working for 6.5 hours without a break
or anything. Especially when I'm on my period and have awful
cramps.”, which should be classified as negative.
Dataset Train: SemEval Test: generated using CrowdFlower
Task #3: Aspect-Based Sentiment Analysis
The output of this task will be a set of aspects of the reviewed
product and a binary polarity value associated to each of such
aspects.
This task requires a set of aspects (such as ‘speaker’,
‘touchscreen’, ‘camera’, etc.) and a polarity value associated with
each of such aspects.
Dataset Train: 5,399 sentences over 2 domains (laptop and
restaurant) Test: 1,097 sentence over 2 domains (laptop and
hotel)
Task #4: Semantic Sentiment Retrieval
This task focuses on the capability of retrieving relevant
documents with respect to opinion-based queries given as input to
participant systems.
Example question: “Documents talking about smartphone
display.”
This task includes: • Information Retrieval (detect features of
given entities) • Named Entity Recognition (detect smartphone
models within the
review possibly using some sort of knowledge base) • Sentiment
Analysis (aggregate features opinions for the entity
sentiment for either overall or feature based retrieval)
Task #5: Frame Entities Identification
This task will evaluate the capabilities of the proposed systems to
identify the objects involved in a typical opinion frame according
to their role: • holders, • topics, • opinion concepts (i.e. terms
referring to highly polarized concepts).
Example: "The mayor is loved by the people in the city, but he has
been criticized by the state government",
an approach should be able to identify that: • “people” and “state
government” are the opinion holders, • “is loved” and “has been
criticized” represent the opinion concepts, • “mayor” identifies a
topic of the opinion and • there are two different opinion
polarities mentioned in the sentence.
Task #6: Subjectivity and Objectivity Detection
Given a text, classify it into objective or subjective. Basically,
an objective sentence does not contain any opinion within it
whereas subjective text does.
The mayor is loved by the people in the city. Subjective
The mayor he has been elected by many voters. Objective
1. INTRODUCTION 2. CHALLENGE TASKS
3. EVALUATION
Evaluation protocol
• Task #1: Precision, Recall, and F-Measure on the inferred
polarity
• Task #3: (i) Precision, Recall and F-Measure on the
Aspect-Extraction; (ii) Accuracy on the polarities inferred on the
correct aspects. Final score: F-Measure* Accuracy
• Most Innovative Approach: taking into account the novelty of the
approach in terms of concept-level analysis and exploitationof
semantics.
Prizes
• Most Innovative Approach: Springer Voucher of 150 Euros
• As there was only one participant to Task #3 (which was also a
participant to Task #1) we removed Task #3 from the
competition
Challenge Participants Task #1 • Mattia Atzeni, Amna Dridi and
Diego Reforgiato: “Fine-Grained Sentiment Analysis on
Financial Microblogs and News Headlines” • Marco Federici: “A
Knowledge-based Approach For Aspect-Based Opinion Mining” • Giulio
Petrucci: “The IRMUDOSA System at ESWC-2017 Challenge on
Semantic
Sentiment Analysis” • Andi Rexha: “Exploiting Propositions for
Opinion Mining” • Walid Iguider and Diego Reforgiato Recupero:
Language Independent Sentiment
Analysis of theShukran Social Network using Apache Spark
Task #3 • Marco Federici: “A Knowledge-based Approach For
Aspect-Based Opinion Mining”
Challenge Participants Task #1 • Mattia Atzeni, Amna Dridi and
Diego Reforgiato: “Fine-Grained Sentiment Analysis on
Financial Microblogs and News Headlines” • Marco Federici: “A
Knowledge-based Approach For Aspect-Based Opinion Mining” • Giulio
Petrucci: “The IRMUDOSA System at ESWC-2017 Challenge on
Semantic
Sentiment Analysis” • Andi Rexha: “Exploiting Propositions for
Opinion Mining” • Walid Iguider and Diego Reforgiato Recupero:
Language Independent Sentiment
Analysis of theShukran Social Network using Apache Spark
Task #3 • Marco Federici: “A Knowledge-based Approach For
Aspect-Based Opinion Mining”
Exploitation Proud to mention that a system that has previously
competed in this challenge is being prototyped by BUP Solution
s.r.l. (http://bupsolutions.com/), Italian company founded by STLAB
of CNR in May 2016
RESEARCH
COMPANIES
SENTIMETRIX INC (USA) http://www.sentimetrix.com/
SentiMetrix tools combine cutting edge NLP technology with advanced
network analysis algorithms implemented on top of highly scalable
architecture. Expert use of modern machine learning techniques
allows the company to achieve highly accurate results in areas
ranging from sentiment diffusion and influence measures to bot
detection, elections results predictions andmental health
evaluations.