Intervention Strategies for Increasing Engagement in Crowdsourcing

1

Intervention Strategies for Increasing Engagement in Crowdsourcing: Platform, Predictions and Experiments

Avi Segal, Ben-Gurion University, Israel

Ya’akov (Kobi) Gal, Ben-Gurion University, Israel

Ece Kamar, Microsoft Research, Redmond WA, USA

Eric Horvitz, Microsoft Research, Redmond WA, USA

Alex Bowyer, University of Oxford, UK

Grant Miller, University of Oxford, UK

Citizen Science (Zooniverse.org)

Wikipedia: scientific research conducted, in whole or in part, by amateur or nonprofessional scientists.

2

Galaxy Zoo

Conclusions

3

The Disengagement Problem

4

Eveleigh, Alexandra, et al. "Designing for dabblers and deterring drop-outs in citizen science.” CHI 2014Sauermann, Henry, and Chiara Franzoni. "Crowd science user contribution patterns and their implications." PNAS 2015

The Approach: Intervention

5

Dimension Description Examples

Target Target population for intervention all, newcomers, heavy users

Time Intervention timing periodical; as soon as risk factor is identified; one day after disengagement

Duration Intervention (interruption) duration for x minutes; until intervention is acknowledged; while user is on webpage

Media Type Method of presentation text; graphics; audio; video; mixed

Channel Medium of delivery email; web page; modal message

Mechanism The method used for this intervention

Messages; help system; task routing; competition; achievements; …

Content Content of the intervention communication

“...If GalaxyZoo didn't suit you, then…”; checkmark image;

Disengagement Prediction Intervention Evaluation

Dimensions: When & What

6

The Intervention Server Platform

7

Predicting Disengagement

8

Predict Disengagement

Intervene

Evaluate

Supervised Learning

Supervised Learning

User will disengage within given window

Session & History

1M records13.5K users

Andrew Mao, Ece Kamar, and Eric Horvitz. "Why stop now? predicting worker engagement in online crowdsourcing.” HCOPM 2013

Intervention Message: Helpful

9


Intervene

Evaluate

Please don’t stop just yet. You’ve been extremely helpful so far. Your votes are really helping us to understand deep

mysteries about galaxies.

Ryan, Richard M., and Edward L. Deci. "Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being." APA(2000)

Intervention Message: Community

10


Intervene

Evaluate

Thousands of people are taking part in the project every month. Visit Talk at

talk.galaxyzoo.org to discuss the images you see with them.

Kraut, Robert E., and Paul Resnick. "Encouraging contribution to online communities." Building successful online communities: Evidence-based social design 2011

Intervention Message: Anxiety

11


Intervene

Evaluate

We use statistical techniques to get the most from every answer; So, you don’t

need to worry about being “right”. Just tell us what you see.

Segal, Avi, et al. "Improving productivity in citizen science through controlled intervention." WWW 2015

Intervention GUI

12


Intervene

Evaluate

Conditions

13


Intervene

Evaluate

# Type

1. No Intervention

2. Random Timing, Helpful Message

3. Random Timing, Community Message

4. Random Timing, Anxiety Message

5. Prediction Based Timing, Helpful Message

6. Prediction Based Timing, Community Message

7. Prediction Based Timing, Anxiety Message

Impact on Contributions

14


Intervene

Evaluate

Impact of History

15


Intervene

Evaluate

Conclusion

Increasing engagement = intervention design + intelligent timing

Code is open sourcedSequential decision making with uncertainty

Off policy policy evaluation Engagement Importance

16

Thank You & [email protected]

Human Aware AI?

30

Learn

Intervene

Evaluate

“Intervention” title 1Papers this morning ~100Learning ~80? 10Intervention? ~10Large scale int. ~2

% 2%

Is this what we want? Is this what we want?