Upload
umair-ul-hassan
View
395
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Copyright 2010 Digital Enterprise Research Institute. All rights reserved.
Digital Enterprise Research Institute www.deri.ie
A CAPABILITY REQUIREMENTS APPROACH FOR PREDICTING WORKER PERFORMANCE IN CROWDSOURCING
Umair ul Hassan, Edward CurryDigital Enterprise Research Institute
National University of Ireland, Galway
9th IEEE International Conference on Collaborative Computing: Networking, Applications and WorksharingAustin, Texas, United StatesOctober 20–23, 2013
Digital Enterprise Research Institute www.deri.ie
Agenda
Motivation Background Task Modelling
Capability Requirements Capabilities Taxonomy
Capability Tracing Experiment Summary
2
Digital Enterprise Research Institute www.deri.ie
Motivation: Heterogeneity
3
Digital Enterprise Research Institute www.deri.ie
WORKER PROFILINGTASK MODELLING
Motivation: Task Routing
Assigning heterogeneous tasks to heterogeneous workers
4
ProfilesModels
TASK ROUTING
MatchingTask↔Worker
ModelsModels
ProfilesProfiles
Digital Enterprise Research Institute www.deri.ie
WORKER PROFILINGTASK MODELLING
Proposal: Performance Prediction
Predict performance of workers on new tasks based on the capabilities required for tasks and assign tasks accordingly
5
ProfilesModels
TASK ROUTING
MatchingTask↔Worker
ModelsModels
ProfilesProfiles
Capability Requirements
Approach
Capability Tracing Model
Performance Prediction
Digital Enterprise Research Institute www.deri.ie
Background: Micro tasks
When micro tasks are crowd sourced Computers cannot do the task Single person cannot do the task Work can be split into smaller tasks
Some online microtask platforms
6
Digital Enterprise Research Institute www.deri.ie
Background: Micro tasks
Most common tasks in Amazon Mechanical Turk (AMT) and CrowdFlower (CFL)
7
Digital Enterprise Research Institute www.deri.ie
Background: Micro tasks
Example of information extraction task in AMT
8
Digital Enterprise Research Institute www.deri.ie
Background: Micro tasks
Example of video transcription task in AMT
9
Digital Enterprise Research Institute www.deri.ie
Task Modelling
Appropriate models are needed to compare and contrast micro tasks.
Capability Requirements approach Capability is defined as the ability of humans to do
things in terms of both the capacity and the opportunity. Four types of capabilities
– Knowledge, – Skill, – Ability, – Other characteristics (e.g. motivation, price, etc)
10
Digital Enterprise Research Institute www.deri.ie
Capability Requirements
Taxonomies have be used to study human task performance, e.g. Fleishman’s taxonomy of human abilities Bloom’s taxonomy of classification of learning
objectives O*NET-SOC taxonomy of occupational classification
We are interested in taxonomy that Describes tasks in terms of human capabilities Helps in comparing tasks in terms of differences and
similarities of capabilities
11
Digital Enterprise Research Institute www.deri.ie
Capabilities Taxonomy
Based on Fleishman’s abilities taxonomy Selected abilities relevant to micro tasks
Comprehension (C): The ability to understand the meaning or importance of something
Bilingualism (B): The ability to speak and understand two languages Writing (W): The ability or capacity to write text in a given language Comparison (M): The ability or capacity to compare things based on
some criteria Judgment (J): The act or process of judging; the formation of an
opinion after consideration Perception (P): The ability or capacity to perceive items visually or
phonetically Identification (I): The process of recognizing something Reasoning (R): The ability to draw conclusions from facts, evidence,
relationships, etc.
12
Digital Enterprise Research Institute www.deri.ie
Requirements of Micro Tasks
13
Digital Enterprise Research Institute www.deri.ie
Capability Tracing
How to model worker’s capabilities? Capability tracing
Inspired by Knowledge Tracing* Estimates probability of a worker knowing a capability
given worker’s responses to test tasks
Worker Profile constrains Set of binary variables representing capabilities Probability estimates of each variable being in a state
14
* A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,” User Modeling and User-Adapted Interaction, vol. 4, no. 4, pp. 253–278, 1994.
Digital Enterprise Research Institute www.deri.ie
Capability Tracing
Probabilistic network of a capability and four parameters of capability tracing model
15
Not Learned
Learned
Correct Incorrect
States of Capability Variable
Values of Response Variable
p(T)
p(L)
p(T): Probability of transition between states
p(L): Probability of a worker learning to employ the capability
p(G): Probability of guess
p(S): Probability of slip
p(G) p(S)
Digital Enterprise Research Institute www.deri.ie
Experiment
Objective Solicit capability requirements of tasks from crowds Evaluation of capability tracing for performance
prediction Three types of micro tasks with manually created
ground truth data Fact verification Image comparison Information Extraction
37 crowd workers including University students Workers from Shorttask.com
16
Digital Enterprise Research Institute www.deri.ie
Crowdsourcing
Custom web application for gathering data Example of fact verification task
17
Digital Enterprise Research Institute www.deri.ie
Capability Requirements of Tasks
Objective 1: Solicit capability requirements of tasks from crowds
18
(a) fact verification (b) image comparison (c) information extraction
Digital Enterprise Research Institute www.deri.ie
Crowd Performance
How the crowd performed on each type of task?
19
Fact Verification task• 37 workers• Best workers perform with
both precision and recall above 0.8
• More variation in recall means some workers were could not spot the incorrect facts
• Ideally tasks should be assigned to workers that lie in the top-right quadrant of the plot
Digital Enterprise Research Institute www.deri.ie
Crowd Performance
Image Comparison (20 workers) and Information Extraction (17 workers)
20
Digital Enterprise Research Institute www.deri.ie
Performance Prediction
Objective 2: Evaluation of capability tracing for performance prediction
Two phases Build model with observation tasks Predict performance on new tasks
21
AC: Consider previous Accuracy as prediction of future performance
CT: Capability Tracing
Digital Enterprise Research Institute www.deri.ie
Summary
Capabilities taxonomy is first steps towards modelling of micro tasks based on human factors
Capability tracing is effective in predicting future performance Even across tasks if there are similar capabilities
Predicted performance can be used to make right task routing decisions
Future Work Evaluate on more types of tasks Evaluate capabilities such as domain knowledge and
skills Define standard tests for measuring worker capabilities
22
Digital Enterprise Research Institute www.deri.ie
Further Reading
U. Ul Hassan and E. Curry, “A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing,” in 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, 2013.
http://deri.ie/users/umair-ul-hassan
23
9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing
Austin, Texas, United StatesOctober 20–23, 2013
Digital Enterprise Research Institute www.deri.ie
Capability Tracing
Conditional probability of worker learning to employ capability p(Ln|On) is calculated
When evidence On is positive
When evidence On is negative
24
Digital Enterprise Research Institute www.deri.ie
Capability Tracing
Probability of worker learning to employ capability
Performance of worker on next task
25