Motivation ML Vandalism...Towards Vandalism Detection in Knowledge Bases: Corpus Construction and...

Motivation

Related workConcentrates on unstructured knowledge bases

Solution IdeaContext Problems

Corpus Construction

SIGIR '15 - Santiago de Chile heindorf@uni-paderborn.de

Towards Vandalism Detection in Knowledge Bases:Corpus Construction and Analysis

Stefan Heindorf Martin Potthast Benno Stein Gregor Engels

Corpus Analysis

Conclusion & Outlook• Vandalism can reduce the quality of knowledge bases• First standardized corpus to study vandalism in structured knowledge bases• Next step: detect vandalism automatically

Downloadour corpus!

Manual Validation1,000 rollbacked revision

Manually reviewed 1,000 undone/reverted revisions1,000 inconspicuous revisions

Option 1: Rollback (by administrators)86 ± 3 %* revisions indeed vandalism

Option 2: Undo/Restore (by all users)62 ± 3 %* revisions indeed vandalism

* 95 % confidence level

Overview of Corpus• 24 million revisions• 103 thousand vandalism revisions

Corpus Revisions over Time

What is vandalized? Who vandalizes?

For our corpus:Rollbacked revisions are considered vandalism

Information systems structuredknowledge bases

Patrollers are busy

Vandalism is notdetected in time

Some people vandalize those knowledge bases MLMachine learning

to detect vandalismVandalismcorpus

Vandalism?Revision 1: yesRevision 2: no

⋮

use

40% of vandalismin structured data

New features beneficial86 % of vandalism

by unregistered users

57%

3%8%

32%

Top vandalized items Top items by category

Unregistered

Registered

Cases Item title47 Cristiano Ronaldo43 Lionel Messi43 One Direction41 Portal:Featured content34 Justin Bieber33 Barack Obama29 English Wikipedia29 Selena Gomez

Contributions• Wikidata Vandalism Corpus WDVC-2015• Corpus analysis

Automatic Revision Labeling• Wikidata revision history• Only non-bot revisions considered• Goal: Automatic labeling as vandalism/non-vandalism

Option 1: Rollback (by administrators)103,205 rollbacked revisions

Option 2: Undo/Restore (by all users)64,820 undone/reverted revisions

1

2

3

4

2

5

revert

20%

20%

16%

14%

13%

9%

8%

1%

12%

21%

9%

15%

8%

4%

31%

1%

Culture

People

Society

Nature

Meta items

Technology

Places

Other

Top 1,000 vandalized itemsTop 1,000 of all items

http://www.uni-weimar.de/en/media/

chairs/webis/corpora/corpus-wdvc-15/