Semantics + Filtering + Search = Twitcident - Exploring Information in Social Web Streams

DelftUniversity ofTechnology

Semantics + Filtering + Search = TwitcidentExploring Information in Social Web StreamsHypertext 2012, Milwaukee, WI – June 28

Fabian Abel, Claudia Hauff, Geert-Jan Houben, Richard Stronkman, Ke Tao

Web Information Systems, TU Delft, the Netherlands

2Semantics + Filtering + Search = Twitcident - Exploring Information in Social Web Streams

200,000,000number of tweets published per day

3

Pukkelpop 2011

People tweet about everything,

everywhere :-)

4

Pukkelpop 2011

81,000 tweets in four hours

became a tragedy

Filtering

200,000,000

Search & Analytics

Useful tweets?

5

Case NijmegenTrain accident


First tweet…

And then your train blasts off full of the anvils. #Nijmegen #veolia


First picture…

Astonishing! My train rams the platform at Nijmegen!

http://pic.twitter.com/QVVfJHyd


Traditional news media

A train ramed the anvils at Nijmegen.


1. (Automatic) Filtering: Given an incident, how can one automatically identify those tweets that are relevant to the incident?

2. Search & Analytics: How can one improve search and analytical capabilities so that users can explore information in the streams of tweets?

Twitter streams

Research Challenges

Filtering

topic

Search & Analytics

information need


Twitcident Pipeline

Automatic Filtering

Search & Analytics


Twitcident system


Twitcident Pipeline

Automatic Filtering

Search & Analytics


Incident detection

• Twiticident relies on Emergency Broadcasting Services for detecting incidents.

• In the Netherlands : P2000 communication network


Incident Profiling• For an incident i:

• The profile of an incident is described as a set of tuples.

• Each tuple includes a facet-value pair (f, v) and its weight to the incident i.

Location, Netherlands

0.4

Incident,Train

accident0.5

Location, Nijmegen

0.8

Orgranization,Veolia

0.6

Incident,Crash

1.0


Twitcident Pipeline

Automatic Filtering

Search & Analytics


Social Media Aggregation • Collecting Twitter messages, pictures, and videos from Social Media Platforms e.g. Twitter, PhotoBucket, Vimeo


Twitcident Pipeline

Automatic Filtering

Search & Analytics


Semantic Enrichment

• Named Entity Recognition

• Classification : Casualties, Damages, Risks…

• Linkage : External Resources

• Metadata extraction


Twitcident Pipeline

Automatic Filtering

Search & Analytics


Filtering

•Which tweets are relevant to the incidents?

• Preprocessing : Language detection

• Semantic Filtering : Compare tweet with P(i)

• Semantic Filtering with News Context• P’(i) : P(i) complemented with f-v pairs from

news


Twitcident Pipeline

Automatic Filtering

Search & Analytics


Faceted Search

•Strategies (ranking)

• Frequency-based

• Time-sensitive based

• Personalized


Real-time analyticsWhat type of things are mentioned in the tweets?

What aspects are mentioned over time? What do people report about over time?

Impact Area


Evaluation - Dataset

• Twitter corpus （ TREC Microblog Track 2011 ） • 16 million tweets (Jan. 24th – Feb. 8th, 2011 )• 4,766,901 tweets classified as English• 6.2 million entity-extractions

• News (Same time period)• 62 RSS News Feeds• 13,959 News Articles• 357,559 entity-extractions


EvaluationFor tweets Filtering (1/2)

Semantic strategies outperform the keyword-based filtering regarding all metrics.


EvaluationFor tweets Filtering (2/2)

The semantic strategy is more robust and achieves higher precisions for complex topics.


EvaluationFor Faceted Search (1/2)

The semantic faceted search strategy improves the search performance by 34.8% and 22.4%.


EvaluationFor Faceted Search (2/2)

The strategies with semantic enrichment outperform the strategy without semantic enrichment in predicting the appropriate facet-values.


Conclusions

• What we have done:

• Twitcident, a framework for filtering, searching, and

analyzing information about incidents that people

publish in their Social Web Streams

• What we have achieved:

• Better filtering of Twitter messages for a given incident.

• Better search for relevant information about an incident

within the filtered messages.


Thank you!

Ke Tao @taubau

@wisdelfthttp://twitcident.org

http://twitcident.org/

Education

Semantics + Filtering + Search = Twitcident - Exploring Information in Social Web Streams