Embed Size (px)
Citation preview
David Karger Sewoong Oh Devavrat Shah
MIT + UIUC
Efficient crowd-sourcing
A classical example
oA patient is asked: rate your pain on scale 1-10o Medical student gets answer : 5o Intern gets answer : 8o Fellow gets answer : 4.5o Doctor gets answer : 6
oSo what is the “right” amount of pain?
oCrowd-sourcingo Pain of patient = tasko Answer of patient = completion of task by a worker
Contemporary example
Contemporary example
Contemporary example
oGoal: reliable estimate the tasks with min’l cost oKey operational questions:
o Task assignmento Inferring the “answers”
Contemporary example
Model a la Dawid and Skene ‘79oN tasks
o Denote by t1, t2, …, tN – “true” value in {1,..,K}
oM workers o Denote by w1, w2, …, wM – “confusion” matrix
o Worker j: confusion matrix Pj=[Pjkl]
o Worker j’s answer: is l for task with value k with prob. Pj
kl
oBinary symmetric case o K = 2: tasks takes value +1 or -1
o Correct answer w.p. pj
Model a la Dawid and Skene ‘79
t1 tNt2 tN-1
w1
w2
wM-1 wM
A11 AN-1 1
AN2
A2M
oBinary tasks:oWorker reliability:
oNecessary assumption: we know
Question
oGoal: given N taskso To obtain answer correctly w.p. at least 1-εo What is the minimal number of questions (edges)
needed?o How to assign them, and how to infer tasks values?
t1 tNt2 tN-1
w1
w2
wM-1 wM
A11 AN-1 1
AN2
A2M
oTask assignment grapho Random regular grapho Or, regular graph w large girth
Task assignment
t1 tNt2 tN-1
w1
w2
wM-1 wM
A11 AN-1 1
AN2
A2M
oMajority:
oOracle:
Inferring answers
t1 tNt2 tN-1
w1
w2
wM-1 wM
A11 AN-1 1
AN2
A2M
oMajority:
oOracle:
oOur Approach:
Inferring answers
t1 tNt2 tN-1
w1
w2
wM-1 wM
A11 AN-1 1
AN2
A2M
o Iteratively learn
o Message-passingo O(# edges) operations
o Approximation ofo Maximum Likelihood
Inferring answers
t1 tNt2 tN-1
w1
w2
wM-1 wM
A11 AN-1 1
AN2
A2M
Inferring answers
t1 tNt2 tN-1
w1
w2
wM-1 wM
A11 AN-1 1
AN2
A2M
o Theorem (Karger-Oh-Shah). o Let n tasks assigned to n workers as per
o an (l,l) random regular graph
o Let ql > √2 o Then, for all n large enough (i.e. n =Ω(lO(log(1/q)) elq)))
after O(log (1/q)) iterations of the algorithm
Crowd Quality
How good?
oTo achieve target Perror ≤ε, we need o Per task budget l = Θ(1/q log (1/ε))
oAnd this is minimax optimal
oUnder majority voting (with any graph choice)o Per task budget required is l = Ω(1/q2 log (1/ε))
no significant gain by knowing side-information(golden question, reputation, …!)
Adaptive solution
Theorem (Karger-Oh-Shah). Given any adaptive algorithm,
let Δ be the average number of workers required per task
to achieve desired Perror ≤ε
Then there exists {pj} with quality q so that
gain through adaptivity is limited
Model from Dawid-Skene ’79
Theorem (Karger-Oh-Shah).
To achieve reliability 1-ε, per task redundancy scales as
K/q (log 1/ε + log K)
Through reducing K-ary problem to K-binary problems
(and dealing with few asymmetries)
Experiments: Amazon MTurk
oLearning similaritieso Recommendationso Searching, …
oLearning similaritieso Recommendationso Searching, …
Experiments: Amazon MTurk
Experiments: Amazon MTurk
Task Assignment: Why Random Graph
Remarks
oCrow-sourcingo Regular graph + message passingo Useful for designing surveys/taking polls
oAlgorithmicallyo Iterative algorithm is like power-iteration
oBeyond stand-alone taskso Learning global structure, e.g. ranking
