Upload
lora-aroyo
View
637
Download
1
Embed Size (px)
Citation preview
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Crowds
Lora Aroyo
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
so9ware systems are ever more intelligent
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
but they don’t actually understand people
focus on human knowledge in machine-‐readable form
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
but there are types of human knowledge that can’t be captured by machines
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
in a form that machines can acquire & process
urge computer science research to push fronCers of human knowledge representaCon
classical AI involves human experts to manually provide training knowledge for machines
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
human expert-‐based ground truth does not scale for current demand for machines to deal with wide
ranges of real-‐world tasks and contexts
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
to provide an approach to capturing human knowledge in a way that is scalable & adequate to real-‐world needs
the key scienCfic challenge is
QuanCty is the new Quality
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
Human ComputaCon adopts human intelligence at scale to improve purely machine-‐based systems
humans accurately perform interpretaCon tasks
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
humans accurately perform interpretaCon tasks
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
can their effort be adequately harnessed in a scienCfically reliable manner that scales across tasks,
contexts & data modaliCes?
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
Research Projects
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
1. extracCng knowledge from social annotaCon
Research Projects
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
1. extracCng knowledge from social annotaCon
2. harnessing of non-‐expert human knowledge
Research Projects
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
1. extracCng knowledge from social annotaCon
2. harnessing of non-‐expert human knowledge
3. nichesourcing of expert knowledge
Research Projects
4. capturing event knowledge
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
1. extracCng knowledge from social annotaCon
2. harnessing of non-‐expert human knowledge
3. nichesourcing of expert knowledge
Research Projects
1. Human Knowledge from Social Tagging
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
Waisda? @ Beeld en Geluid – an online video tagging game, e.g. NCRV and VARA – power of the crowd for Web-‐scale enrichment of A/V collecAons
– significant accuracy for improving IR beyond the state-‐of-‐the-‐art of expert-‐based annotaAons
– sustainable means for engaging large volunteer crowds
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
1. Human Knowledge from Social Tagging
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
hZp://waisda.nl/ hZp://spotvogel.vroegevogels.vara.nl/
Engage with Games in Entertainment
Following the grandeur of Baroque, Rococo art is often dismissed as frivolous and unserious, but Waldemar Januszczak disagrees. […] The first episode is about travel in the 18th century and how it impacted greatly on some of the finest art ever made. The world was getting smaller and took on new influences shown in the glorious Bavarian pilgrimage architecture, Canaletto's romantic Venice and the blossoming of exotic designs and tastes all over Europe.
Rococo: Travel, pleasure, madness
A boarding school where boys from the Dutch East Indies receive a vocaAonal educaAon has been set up by the ministry of Social Affairs near Batavia. Trumpeter sounds the reveille; -‐ the boys get out of bed; -‐ the muster is held in front of the building; aIerwards the boys stand in line with a food bowl; they get rice pudding and eat this around long tables in the open air; -‐ the boys get instrucAons and pracAce a.o. drawing shapes, forging, metalworking, filing and woodworking;
News from Indonesia: youth care
hZp://vista-‐tv.eu hZp://dive.beeldengeluid.nl
Crowdsourcing for Video Analysis
2. Harnessing Non-‐expert Human Knowledge in Medical Domain
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
2. Harnessing Non-‐expert Human Knowledge in Medical Domain
Training IBM Watson – gathering higher quality ground truth data from the crowd than from experts
– open source human-‐machine computaAon framework for accurately measuring quality of resulAng data
– CrowdTruth.org
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
http://CrowdTruth.org http://data.CrowdTruth.org/ http://game.crowdtruth.org
Achieving Expert-Level Annotation Quality with the Crowd
The Case of Medical Relation Extraction
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
Crowdsourcing for Text Analysis
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
hNp://game.crowdtruth.org
Engage with Games in Sciences
3. Nichesourcing Expert Knowledge
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
3. Nichesourcing Expert Knowledge
Cultural Heritage – Different layers and domains of experAse – Intrinsic moAvaAon of knowledgeable crowds – Accurator.nl @ Rijksmuseum
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
seman'c enrichment with LOD can be complemented with domain knowledge from niches of experts
there are massive niches of experts online …
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
hNp://annotate.accurator.nl/
Engage the Niche in Cultural Heritage
hNp://accurator.nl/
Engage with Games in Cultural Heritage
4. Capturing Event Knowledge
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
4. Capturing Event Knowledge
interpretaCon support framework Digital HermeneuCcs – text, images and videos, e.g. newspapers, tweets, TV and radio programs, and images
– events carry ambiguity, bias & wide range of perspecAves – knowledge representaAon challenges, e.g informaAon granularity, temporal and provenance modeling
– Agora, DIVE+ @ KB, Beeld en Geluid, Amsterdam Museum – Crowddriven.nl for TV show Game of Thrones
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
engaging users
through event
narratives
“Digital Hermeneu/cs: Agora and the online understanding of cultural heritage” In proceedings of Web Science Conference, (ACM: New York,
2011)
InterpretaCon Support for Scholars
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
Browsing support for wide audience
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
Engage with Games in Entertainment
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
understanding perspecCves – diversity of opinions from the crowd – mulAtude of contexts – ambiguity in language, images or videos – independent interpretaCons – aggregated view – the big picture
– a new approach to understanding semanCcs by harnessing the power & diversity (disagreement on the correct interpretaAon) of the crowd
– human disagreement is essenCal in helping machines with semanCc interpretaCon
Goodbye to Truth
Hello Perspectives
Goodbye to Truth
Human Computation
making sense of human
semantics
Goodbye to Truth
Human Computation
making sense of human
semantics
http://lora-aroyo.org http://slideshare.net/laroyo
@laroyo