Upload
multimediaeval
View
202
Download
0
Embed Size (px)
Citation preview
Work Like a BeeTaking Advantage of Diligent Crowdsourcing Workers
Presentation: Michael Riegler, Simula Research Laboratory, Norway Preben N. Oslen, Simula Research Laboratory Pål Halvorsen, Simula Research Laboratory [email protected]
The Idea• Use “good” workers
• Like the task or good with it
• Will do it much better
• Diligent workers
• Do most of the work
• Extra: additional crowd workers
The Runs
• 4 different runs
• Run 1: MJV with additional crowdsourcing
• Run 2: Diligent Workers vote only
• Run3: MJV with weighted diligent workers
• Run4: Run3 with additional crowdsourcing
The ResultsRun WF1 score True Labels
Predicted Labels
R1 0.7207 183, 63, 291 192, 68, 279
R2 0.6919 183, 63, 291 208, 95, 234
R3 0.6912 183, 63, 291 208, 87, 242
R4 0.6919 183, 63, 291 208, 95, 234
Baseline 0.3809 183, 63, 291 0, 0, 537
The Analysis
• Every run is better than the baseline
• R1 is the best performing run
• Difference between R1 and R2 shows that diligent workers perform better
• Can for example be used to save costs
• R2, R3 and R4 perform similar
• Additional Crowdsourcing does not improve results
The Conclusion
• Very simple approach
• Showed that it can work
• No gain trough additional crowdsourcing
• Extending work on larger datasets
Tusen takk og ha det bra!
The End