21
The future of crowd work Aniket Kittur et al., (2013). The future of crowd work, CSCW’13 Mejdl Safran Fall 2013 SIUC

The future of crowd work Aniket Kittur et al., (2013). The future of crowd work , CSCW’13

  • Upload
    aric

  • View
    95

  • Download
    0

Embed Size (px)

DESCRIPTION

The future of crowd work Aniket Kittur et al., (2013). The future of crowd work , CSCW’13. Mejdl Safran Fall 2013 SIUC. More complex, creative and highly valued work. - PowerPoint PPT Presentation

Citation preview

Page 1: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

The future of crowd workAniket Kittur et al., (2013). The future of crowd work, CSCW’13

Mejdl SafranFall 2013

SIUC

Page 2: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

More complex, creative and highly valued work

“a platform is needed for managing pools of tasks and workers. Complex tasks must be decomposed into smaller subtasks, each designed with particular needs and characteristics which must be assigned to appropriate groups of workers who themselves must be properly motivated, selected (e.g., through reputation), and organized (e.g., through hierarchy). Tasks may be structured through multi-stage workflows in which workers may collaborate either synchronously or asynchronously. As part of this, AI may guide (and be guided by) crow workers. Finally, quality assurance is needed to ensure each worker’s output is of high quality and fits together.”

Page 3: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Research challenges in 12 major areas

1. The future of crowd work processes2. The future of crowd computation3. The future of crowd workers

Page 4: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Work: Workflow

Motivation/Goals Complex crowd work cannot accomplished by simple parallel

approaches (e.g., aggregating multiple independent judgments) Workflows are needed that facilitate decomposing tasks into

subtasks, managing the dependencies between subtasks, and assembling the results.

Page 5: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Work: Workflow

Research Proposal Improving existing workflows toward more general tasks and complex

problems that have no clearly defined solution (creativity, civic planning, etc.)

Comparing to simple workflow, crowd workflow could involve a much larger scale of operation, an a much more heterogeneous set of actors

How to validate? We must experiment on a large space of parameters, incentives and

decompositions. Costs of doing so can be reduced through models of worker behavior,

or by reusing proven design patterns..

Page 6: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Work: Task Assignment

Motivation/Goals Sharing limited resources (workers) requires coordination. While workers are continuously employed with tasks that

match their interests, requesters will see their tasks completed quickly.

ESP game and Galaxy Zoo use FCFS model oDesk and AMT use market model

Page 7: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Work: Task Assignment

Research Proposal Data partitioning, OS scheduling and failover! Or new theoretical model. “Several workers in our survey complained that they spent too

much energy finding appropriate tasks” (Task Recommendation).

Research question: should tasks be pulled by workers or pushed by platforms?

Page 8: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Work: Quality Control

Motivation/Goals Why? High throughput, low transaction costs ands

complex/subjective tasks. Cheating or game the system. Workers with low expertise and requesters with unclear

instructions.

Page 9: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Work: Quality Control

Research Proposal Up-front task design: peer-review, agreement filters (writing an

essay!!), optimize instructions, etc. Post-hoc result analysis: gold standard data (prevent biases from

dominating the results) and workers’ votes (costs more). Better approach: use ML to predict the quality of a worker’s

output from their behavior. Open-ended work and highly skilled tasks?? Is it possible to robustly measure worker’s skills at tasks such as

audio engineering and poetry? Should we rely on peer evaluation?

Page 10: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Work: Real-time Crowd Work

Motivation/Goals We need to create flash crowds: groups of individuals who

arrive moments after a request and can work synchronously. All on-demand CS are limited by the crowd latency.

Page 11: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Work: Real-time Crowd Work

Research Proposal Fast recruitment is a major research area in real-time CS. Time-limited tasks (e.g., search for a missing person and timed

competitions) E-mail a set of workers the night before the study and

announced a time for the experiment. Paying workers a small wage to stay on call. They need to be modeled using queuing theory. Scaling up to increased demand for real-time workers Making workers efficient enough to collectively generate

results ahead of time deadlines

Page 12: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Computation: Crowds Guiding AIs

Motivation/Goals In human computation, people act as computational

components and perform the work that AI systems lack the skills to complete.

By tapping into crowd intelligence, computational systems can support a much broader set of tasks

Crowd workers may end up training machines to replace them!!

Gathered data is used to train algorithms (e.g., Google image search engine, birds count project)

Page 13: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Computation: Crowds Guiding AIs

Research Proposal Integrating crowds more deeply into algorithms. Design ML algorithms that more deeply understand the human

nature of these labels.

Page 14: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Computation: AIs Guiding Crowds

Motivation/Goals Humans can introduce errors, and errors are amplified as they

propagate through the crowd. Integrate crowds inside of software and use the software to

help guide crowd work. E.g., ML model can determine which work products may still

be improved and then assign workers most likely to make such improvements.

Page 15: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Computation: AIs Guiding Crowds

Research Proposal AIs can point out what others have done in similar contexts. We need to move from a setting where simple AIs completely

determine a workflow to a richer, mixed setting where crowds and AIs jointly teach each other, and jointly control the work process.

Page 16: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Computation: Crowdsourcing Platforms

Motivation/Goals Limiting ourselves to exiting platforms greatly restricts the

scope and nature of change we can enact.

Page 17: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Computation: Crowdsourcing Platforms

Research Proposal Different platforms for different aims. MobileWorks (mClerk): through SMS, more populations. Vending machine with purpose (exam grading) targeting

experts. Labor exchanged policy issues How can platforms disclose enough info to be trusted as a

source of worker quality while also maintaining privacy? Security concerns both attacks on platforms or use of

platforms for launching attacks

Page 18: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Workers: Reputation

Motivation/Goals “Graduated from UC Berkeley and working Google” Traditional employers judge a prospective employee's

education and work history through interviews, transcripts, and references.

“I have an accepting rate of 4 out of 5 on AMT”

Page 19: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Workers: Reputation

Research Proposal We want better reputation rankings for workers to be

established with the platforms Reputation systems must be robust to cheating and gaming

while supporting low-transaction cost hiring. One way: a web of trust! (workers and requesters validate each

other. What if malicious build their own web of trust!!! (gaming the

system) Technical tools for sharing information about workers

requester abuses!! (privacy)

Page 20: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Workers: Motivation & Rewards

Motivation/Goals Crowd workers are a diverse and multifaceted population with

a range of motives and experiences Worker said “Mturk should encourage and reward requesters

that provide clear instructions, quick payment, and higher pay”

Page 21: The future of crowd  work Aniket Kittur  et al., (2013).  The future of crowd work ,  CSCW’13

Crowd Workers: Motivation & Rewards

Research Proposal Past research:

Requesters should clearly understand and communicate desired behaviors

Understand and worker motivations and incentives with these desired behaviors

Design the requests and incentive structures in order to achieve both effective task completion and worker satisfaction

Beyond financial incentives Build systems that support workers’ diverse motivations.