View
5.331
Download
1
Category
Tags:
Preview:
DESCRIPTION
This talk was given as a keynote at the I-Semantics 2012 conference, Graz, Austria
Citation preview
Dial E for
Events
Lora Aroyo
Monday, September 10, 12
Observationevents are important
events are omni-presentevents carry different points of viewin the world, e.g. news, science etc.
in our personal lives, e.g. social
networking
iSemantics2012 Lora Aroyo @laroyoFlickr: elkabong Monday, September 10, 12
Position
The human disagreement & vagueness of events are part of
the event semantics
iSemantics2012 Lora Aroyo @laroyoFlickr: elkabong Monday, September 10, 12
Objects vs. Events
events perdure = their parts exist at different time pointsobjects endure = they have all their parts at all points in time
objects are wholly present at any point in time, events unfold over time
iSemantics2012 Lora Aroyo @laroyoFlickr: vanilllaph Monday, September 10, 12
Events are importantcreate context for objects, e.g. people, locations, organizations, etc.
Lora Aroyo @laroyo iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events are importantcreate meaning for objects, e.g. artifacts, pictures, videos.
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events are importantlink concepts, objects, and stories.
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events in the Worldevents anchor the information we
consume daily
iSemantics2012 Lora Aroyo @laroyoFlickr: craftydogma Monday, September 10, 12
Events @ Google News
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events @ Google News
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Julian Assange’s Extradition Row
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Julian Assange’s Extradition Row
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Julian Assange’s Extradition Row
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Julian Assange’s Extradition Row
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Lance Armstrong’s Doping Fight
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Lance Armstrong’s Doping Fight
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
The Arab Spring
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
The Arab Spring
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events @ Social Web
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events @ Social Web
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events @ Social Web
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events @ Social Web
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events @ Social Web
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Events are VagueHumans have no clear notion of what events are
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
“event is a significant "happening" or gathering of people. I would define a "happening" as an event if the group of people gathered were united in one common goal.”
We Asked the Crowd What an EVENT isiSemantics2012 Lora Aroyo @laroyoFlickr: massimo vitali
Monday, September 10, 12
We Asked the Crowd What an EVENT is
Event is a happening, which can be scheduled or unscheduled. An earthquake or fire happens (unscheduled). A wedding or birthday party (scheduled). It is an occasion that is unusual and tends to be memorable.
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
“An event would be any occurrence where physical action has taken place. It may be a single, momentary instance (I sneezed), or it may span a period of time (the festival ran for four hours). An event may also be made up of a number of smaller events, such as a day at school is an event, but each individual class is also an event itself. Basically an event must have a physical action over any delimited time span.”
We Asked the Crowd What an EVENT isiSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
“A planned public or social get together or occasion.”
“an event is an incident that's very important or monumental”
“An event is something occurring at a specific time and/or date to celebrate or recognize a particular occurrence.”
“a location where something like a function is held. you could tell if something is an event if there people gathering for a purpose.”
“Event can refer to many things such as: An observable occurrence, phenomenon or an extraordinary occurrence.”
We Asked the Crowd What an EVENT isiSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
What do Experts think an EVENT is?
“an event is the exemplification of a property by a substance at a given time” Jaegwon Kim, 1966
“events are changes that physical objects undergo” Lawrence Lombard, 1981
“events are properties of spatiotemporal regions”, David Lewis, 1986
iSemantics2012 Lora Aroyo @laroyounder30ceo.comMonday, September 10, 12
Event-centric Projectshow events can be detected & extracted from
natural language text
how those extracted events are represented for use on the semantic web
how to identify the same events in different sources
how to capture different perspectives
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Activists
prominent on the new web through different channels
by nature multi-perspective, biased & emotional
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Activists
prominent on the new web through different channels
by nature multi-perspective, biased & emotional
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Mapping Online Networks of Activism
“All protest events Greenpeace participated in.”
blogs, news, activists websites
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Mapping Online Networks of Activism
“All protest events Greenpeace participated in.”
• build visualizations
• create appropriate analytics
• answer questions of end users & social scientists
blogs, news, activists websites
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
• What happened before/after?
• Who does what, when, and where?
• All bomb attacks in the 1950s
• In what events did Indonesia participate?
• ‘Grand narratives’
• objects (digitized artworks and artifacts)
• events (concrete particulars)
• entities (actors, locations, periods)
• narratives (organization of events)
Extracting Historical Events
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
• generate meaningful event sequences• capture the different perspectives• serve both end users & history researchers
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Timelines from Text
4 right, 2 wrong, 3 missing eventstwo have no explicit times & are in the
wrong orderOne involved al-Qaeda but took place in
Jordan on the Syrian border
does a fuzzy task require fuzzier metrics?
“al-Qaeda activities in Syria”
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Timelines from Text
4 right, 2 wrong, 3 missing eventstwo have no explicit times & are in the
wrong orderOne involved al-Qaeda but took place in
Jordan on the Syrian border
does a fuzzy task require fuzzier metrics?
“al-Qaeda activities in Syria”
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Machine Readingbuild event timelines from text in 2 example domains
• NFL: news articles on football;
• Ontology: 4 classes, 20 relations
• Intel: news articles on terrorist events;
• Ontology: 20 Classes, 50 relations
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Why is event semantics hard?
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
According to NLP traditionGather
your source material
1Extract
events and properties
2
Analyze
find links between events
3
Visualize
statistics, timelines, etc.
4
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
but for events we stumble Gather
your source material
1Extract
events and properties
2
Analyze
find links between events
3
Visualize
statistics, timelines, etc.
4
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
but for events we stumble Gather
your source material
1Extract
events and properties
2
Analyze
find links between events
3
Visualize
statistics, timelines, etc.
4
experts typically:
define a problemannotate ground truth
train evaluate
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Closed World Dictatorship
1. domain experts define the meaning2. using limited vocabulary
3. aim for agreement
to fix the problem of high disagreement for eventsexperts enforce more tyranny - stricter rules
comparatively little annotated data for training & evaluation of eventdetection systems
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
But the World is Open
1. events have multiple dimensions2. each dimension has levels of granularity
3. people have different views on both
all this leads to very complex semantics
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
and our goal is ...
1. not to enforce agreement2. to capture different view points
3. to teach machines to reason in the disagreement space
iSemantics2012 Lora Aroyo @laroyoFlickr: elkabongMonday, September 10, 12
PositionArtificially restricting humans does not help machines to learn.
Machines will learn from diversity
iSemantics2012 Lora Aroyo @laroyoFlickr: elkabongMonday, September 10, 12
• Museum, libraries, archives & researchers have been dominating the views.
• Controlled vocabularies & annotation schemes were leading.
• Professionals enforced agreement among themselves.
• End-users needs & tasks are not considered.
Professional Dictatorship of the Closed World
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
there is a tiny overlap between end-user terminology & professional annotations
the latter are typically coarse-grained & refer to entire object / topic
iSemantics2012 Lora Aroyo @laroyoFlickr: ganzelkaMonday, September 10, 12
only 1,900 tags (32,200 in total) match in vocabularies 257 in people (83 validated) 1,661 in geo (666 validated)
9,796 validated, but no match in professional vocabulary8% professional vocab
23 % lexical vocab 63% meaningful Google matches
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Amateur democracy of the Open World
• Once the Web opened the world of information, professional dictatorship clashed with end-users democracy.
• What professionals consider interesting, relevant or important does not match what users think of it.
• Amateurs cannot find what they were searching for.iSemantics2012 Lora Aroyo @laroyo(c) banksy
Monday, September 10, 12
people are interested in different annotation categories than the professionals
iSemantics2012 Lora Aroyo @laroyoFlickr: ganzelkaMonday, September 10, 12
Video aspects that are described by those tags:
non-visual (0)perceptual (11), e.g. color
conceptual (1,332)
Tag sample: 1,343 verified tags of 5 random video fragments
195 tags (adverbs & adjectives) couldn’t be classified
Abstract General Specific Total
Who 10 1665
17712
31%
What 73 563 12 57%
Where 0 6831
8 7%
When 4 31 6 5%
Total 7% 74% 9%
Object tags (1,313)Scene tags (30)
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Harnessing the Crowd
• we include the user’s opinion as first class citizens.
• this brings the need to combine all these (different) opinions into a system of opinions that makes sense.
• new solutions are needed, e.g. crowdsourcing of perspectives on events that exploit disagreement
iSemantics2012 Lora Aroyo @laroyoFlickr: AmyJanelleMonday, September 10, 12
What do People Disagree on?
are sub-events always mere parts?are “mentions” meaningful for events?
are events coreferential across documents? (e.g. perspectives, observations)
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
the bombing targeted a housing development in Baghdad, killing 3 and injuring 13
indistinguishable by people, confusable: is bombing part of killing, or killing part of bombing?
What about targeting?
“merelogically extensional” (i.e arbitrary): container bursting into fragments as a result of explosion
some events don’t exist: an action by military forces prevented the bombing.
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Disagreement Framework• ontology: disagreements on the basic status of events
themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.
• granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.
• interpretation: disagreements that result from (non-granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September.
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Disagreement Framework• ontology: disagreements on the basic status of events
themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.
• granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.
• interpretation: disagreements that result from (non-granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September.
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Granularity Disagreement
spatial, temporal, participants
compositional, classificationaliSemantics2012 Lora Aroyo @laroyo
Monday, September 10, 12
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Event Participants Disagreement
Prime minister Benjamin
Netanyahu
Benjamin Netanyahu
Israeli Prime minister
Cabinet
Benjamin Netanyahu’s
Cabinet
Israeli Cabinet
his Cabinet
Israeli Government
{TOLD}
50%
35%
15%
10%
15%
5%
45%
15%
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Temporal Disagreement
Spring 1998
March 1, 1998
March 1998
SundayPrime minister
Benjamin Netanyahu
Benjamin Netanyahu
Israeli Prime minister
{TOLD}
50%
35%
15%
25%
15%
50%
5%
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Spatial Disagreement
Lebanon
IsraelSouthern Lebanon
Israel's Northern Frontier
Middle East
{WILLING TO WITHDRAW}
35%
45%
10%
30%
65%
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Approach Principles1. tolerate, capture & exploit disagreement
2. understand the range of disagreements by creating a space of possibilities with frequencies & similarities
3. score the machine output based on where it falls in this space4. adaptable to new annotation tasks
iSemantics2012 Lora Aroyo @laroyoFlickr: auroilleMonday, September 10, 12
it seems to refer to an inference or communicated feeling more than specific event.
does not refer to an event
a group of people did something specific at a specific point in 6me.
refers to an event
the actors in ques6on (top Israeli officials) performed an ac6on during a specified 6me (Sunday).
refers to an event
it refers to what the israelis did on sunday, a specific 6me.
Top Israeli officials SENT strong new SIGNALS Sunday that Israel wants to withdraw from southern Lebanon, ...
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Because it is describing a historical issue concerning the resolu6on of 1978
refers to an event
That 1978 resolu6on calls for Israel's uncondi6onal WITHDRAWAL from the self-‐declared security zone it occupies in south Lebanon, ...
it is not a par6cular movement that has or is going on but a request that the country of Israel remove their forces from the zone they occupy.
does not refer to an event
the sentence is speaking of a demand for a withdrawal that had not yet occurred.
does not refer to an event
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
The Dark Side of Crowdsourcing Disagreement
• disagreement is beautiful, except when it results from spamming• crowdsourcing has to account for people that want to get paid for
not doing any work• spammers generate disagreement for the wrong reasons• most spam detection requires gold standard
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Spam or not?
• cut & paste from text• identical to other explanations • much shorter time than the average• low trust value of the worker• shorter than 5-6 words
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Spam or not?Prime Minister Benjamin Netanyahu TOLD his Cabinet on Sunday that Israel was willing to ...
Because being told something doesn't seem like an event.
does not refer to an event
Because the WAR is being described as a costly event.
refers to an event
Top Israeli officials sent strong new signals Sunday that Israel wants to withdraw from southern Lebanon, where a costly WAR of aTri6on has been claiming soldiers' lives.
Because Israel WANTS TO WITHDRAW from Lebanon.
refers to an event
Top Israeli officials sent strong new signals Sunday that Israel WANTS TO WITHDRAW from southern Lebanon, ...
+ low worker trust
+ low worker trust
+ low worker trust
Because WANTS TO WITHDRAW is an ac6on.
refers to an event
Top Israeli officials sent strong new signals Sunday that Israel WANTS TO WITHDRAW ...
+ short time
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Motivation-Verification Method
• 2-stage method:• disagreement collection + motivation• spam filtering = motivation judgement
• Additionally:• sample the motivation stage to manually
extract gold standard for stage 2
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
Phase I:A. Collect event annotations +
motivations
Extraction of Putative Events
input: putative events
Phase I:C. Filtering spam
event annotations
input: output of A
Phase II:A. Collect event types
+ motivations
input: list of events
Phase II:B. Filtering spam
event types
input: output of A
Manual selection of Gold Questions
input: output of A
Manual selection of
Gold Questions
input: output of A
Phase III:A. Collect event
modalities + motivations
input: list of events
Phase III:B. Filtering spam event modalities
input: output of A Manual
selection of Gold Questions
input: output of A
Phase IV:A. Collect event
role fillers + motivations
Phase IV:B. Filtering spam event role fillers
input: output of A Manual
selection of Gold Questions
input: output of A
input: list of events
• a new way of measuring ground truth
• a new set of semantic features for learning in event extraction
iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12
PositionArtificially restricting humans does not help machines to learn.
Machines will learn from diversity
iSemantics2012 Lora Aroyo @laroyoFlickr: elkabongMonday, September 10, 12
Position
The human disagreement & vagueness of events are part of
the event semantics
iSemantics2012 Lora Aroyo @laroyoFlickr: elkabongMonday, September 10, 12
finally ...
end the tyranny
disagree
ment is beautifu
l
iSemantics2012 Lora Aroyo @laroyoFlickr: elkabongMonday, September 10, 12
Acknowledgements
Michiel Hildebrand
Lotte Belice Baltussen
Jacco van Ossenbruggen
Marteen Brinkerink
Johan OomenGuus Schreiber
Riste Gligorov
Marieke van Erp
Lourens van der Meij
Roxane Segers
Piek Vossen
Susan Legêne
Thomas Ploeger
Chiel van den Akker Frank de Bakker
Bibiana Armenta
Iina Hellsten
Geertje Jacobs
Geert-Jan Houben
Chris Welty
Monday, September 10, 12
Questions?
@laroyohttp://lora-aroyo.org
Monday, September 10, 12
Recommended