SOCIAL MACHINESTECHNOLOGY DESIGN AND INCENTIVES

Elena Simperle.simperl@soton.ac.uk @esimperlJuly 4th, 2016

1

TUTORIAL OVERVIEW

Social machines as engine of creativity, growth, and sociality

This tutorial covers Introduction to social machines How-to crowdsourcing design How-to study citizen science projects Examples from current research

TUTORIAL@ISWC2013

THE ORDER OF SOCIAL MACHINES

Real life is and must be full of all kinds of social constraint the very processes from which society arises. Computers can help if we use them to create abstract social machines on the Web: processes in which the people do the creative work and the machine does the administration []

The stage is set for an evolutionary growth of new social engines.

Tim Berners-Lee, Weaving the Web, 1999

3

PEOPLE SUPPLY, CURATE AND ANALYZE DATA

OR CREATE CONTENT FROM SCRATCH

FOR FREE OR AGAINST A REWARD

PEOPLE ARE (ELEMENTARY) PROBLEM SOLVERS

SOMETIMES ON ANY GIVEN TOPIC

PEOPLE LIKE TO SHARE THEIR VIEWS

9

SOMETIMES THEY WORK TOGETHER

10

SOMETIMES THEY COMPETE

11

IT DOESNT GO WELL ALL THE TIME

12

BUT THE ONES THAT DO WELL SEEM TO HAVE THINGS IN COMMON

i. Problems solved by the scale of human participation

ii. Timely mobilisation of people, technology, and information resources

iii. Access to (or else the ability to generate) large amounts of relevant data

iv. Confidence in the quality of the data

v. Efficient, effective and equitable

vi. Trust in the agents and process

vii. Aligned incentives and network effects

viii. Intuitive, user-centred interfaces

ix. Cross platform support

x. Exploit the power of the open

sociam.org

@project_sociam

WHAT IS A SOCIAL MACHINE? Exercise 115

Based on the definition and examples mentioned so far, discuss examples of social machines you are familiar with.

When is a social machine successful?

How do social machines compare with: human computation, crowdsourcing, collective intelligence, social computing, wisdom of the crowds, open innovation, socio-technical systems etc.?

16

SOCIAL MACHINES AS CROWDSOURCING

17

CROWDSOURCING:PROBLEM SOLVING VIA OPEN CALLS"Simply defined, crowdsourcing represents the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. This can take the form of peer-production (when the job is performed collaboratively), but is also often undertaken by sole individuals. The crucial prerequisite is the use of the open call format and the large network of potential laborers.

[Howe, 2006]

7/4/2016 18

DIFFERENT FORMS AND PLATFORMS TO CHOOSE FROM

19

Macrotasks

Microtasks

Challenges

Self-organized crowds

Crowdfunding

Source:

[Prpi et al., 2015]

FOR EXAMPLE: MICROTASK CROWDSOURCING

7/4/2016 20

Work is broken down into smaller pieces that can be solved independently

Are social machines just crowdsourcing?

CROWDSOURCING & SOCIALITY

CROWDSOURCING CREATIVE TASKS

CROWDSOURCING & ETHICS

[Difallah et al, 2015]

MANY QUESTIONS TO ANSWER

TASK DESIGNWORKFLOW DESIGN AND EXECUTION

TASK INTERFACES QUALITY ASSURANCE

TASK ASSIGNMENT

CROWD TRAINING AND

FEEDBACKINCENTIVES

ENGINEERING

COLLABORATION, COMPETITION,

SELF-ORGANIZATION

REAL-TIME DELIVERY NICHESOURCING

EXTENSIONS TO TECHNOLOGIES

SOCIAL MACHINES

ENGINEERING

LUNCH BREAKe.simperl@soton.ac.uk@esimperl

sociam.org26

CROWDSOURCING DESIGN27

CROWDSOURCING DESIGN 101

What: Goal

Who: Contributors

How: Process

Why: Motivation and incentives

http://sloanreview.mit.edu/article/the-collective-intelligence-genome

28

Image at http://jasonbiv.files.wordpress.com/2012/08/collectiveintelligence_

genome.jpg

http://sloanreview.mit.edu/article/the-collective-intelligence-genome/http://jasonbiv.files.wordpress.com/2012/08/collectiveintelligence_genome.jpg

EXAMPLE: ZOONIVERSE

WHAT IS OUTSOURCED

Object recognition, labeling, categorization in media content

WHO IS THE CROWD

Anyone

HOW IS THE TASK OUTSOURCED

Highly parallelizable tasks

Every item is handled by multiple annotators

Every annotator provides an answer

Consolidated answers solve scientific problems

WHY DO PEOPLE CONTRIBUTE

Fun, interest in science, desire to contribute to science

7/4/2016 29

zooniverse.org

DESIGN YOUR OWN Exercise 230

IN THIS TALK: CROWDSOURCING DATA CITATIONThe USEWOD experiment Goal: collect information about the usage of Linked

Data sets in research papers Explore different crowdsourcing methods Online tool to link publications to data sets (and

their versions) 1st feasibility study with 10 researchers in May

2014

7/4/2016 31

http://prov.usewod.org/

9650 publications

Discuss how you would crowdsource the problem illustrated on the previous slide.

Consider using the schema presented earlier.

Think about how you would ensure that the data is accurate.

(15 minutes discussions in groups, then plenary discussion of results)

32

DIMENSIONS OF CROWDSOURCING33

WHAT

In general: something you cannot do (fully) automatically

In our example: annotating research papers with data set information Alternative representations of the domain What if the paper is not available? What if the domain is not known in advance or is infinite? Do we know the list of potential answers? Is there only one correct solution to each atomic task? How many people would solve the same task?

7/4/2016 34

What Who

How Why

BROAD RANGE OF TASKS

7/4/2016 35

What Who

How Why

WHOIn general An open (unknown) crowd Scale helps Qualifications might be a pre-requisite

In our example People who know the papers or the data sets Experts in the (broader) field Casual gamers Librarians Anyone (knowledgeable of English, with a computer/cell phone etc) Combinations thereof

7/4/2016 36

What Who

How Why

Country of residence

United States: 46.80% India: 34.00% Miscellaneous: 19.20%

7/4/2016 37

A LARGE, BUT NOT ALWAYS DIVERSE CROWD

What Who

How Why

SIGNIFICANT AMOUNT OF RESOURCES AND TIMELY DELIVERY

7/4/2016 38

What Who

How Why

SIGNIFICANT AMOUNT OF RESOURCES AND TIMELY DELIVERY (2)

39

http://www.mturk-tracker.com/

60% of workers spend more than 4 hours a week on MTurk

What Who

How Why

HOW: PROCESSIn general: Explicit vs. implicit participation

Affects motivation

Tasks broken down into smaller units undertaken in parallel by different people Does not apply to all forms of crowdsourcing

Coordination required to handle cases with more complex workflows Sometimes using different crowds for workflow parts

Example: text summarization

Task assignment to match skills, preferences, and contribution history Example: random assignment vs meritocracy vs full autonomy

Partial or independent answers consolidated and aggregated into complete solution Example: challenges (e.g., Netflix) vs aggregation (e.g., Wikipedia)

7/4/2016 40

See also [Quinn & Bederson, 2012]

What Who

How Why

HOW: PROCESS (2)In our example: Use the data collected here to train a IE algorithm Use paid microtask workers to go a first screening, then expert crowd to sort out

challenging cases What if you have very long documents potentially mentioning different/unknown

data sets? Competition via Twitter

Which version of DBpedia does this paper use? One question a day, prizes Needs golden standard to bootstrap and redundancy

Involve the authors Use crowdsourcing to find out Twitter accounts, then launch campaign on

Twitter Write an email to the authors

Change the task Which papers use Dbpedia 3.X?

Competition to find all papers

7/4/2016 41

What Who

How Why

HOW: VALIDATION In general: Solutions space closed vs. open

Redundancy vs. iterations/peer-review Performance measurements/ground truth

Available and accepted Automated techniques employed to predict accurate solutions

In our example: Domain is fairly restricted

Spam and obvious wrong answers can be detected easily When are two answers the same? Can there be more than one correct answer per question?

Redundancy may not be the final answer Most people will be able to identify the data set, but sometimes the actual

version is not trivial to reproduce Make educated version guess based on time intervals and other features

7/4/2016 42

What Who

How Why

WHYMoney (rewards)

Love

Glory

Love and glory reduce costs

Money and glory make the crowd move faster

Intrinsic vs extrinsic motivation

Rewards/incentives influence motivation. They are granted by an external judge

Successful volunteer crowdsourcing is difficult to predict or replicate Highly context-specific Not applicable to arbitrary tasks

Reward models often easier to study and control (if performance can be reliably measured) Not always easy to abstract from social

aspects (free-riding, social pressure) May undermine intrinsic motivation

43

What Who

How Why

WHY (2)

In our example:

Who benefits from the results

Who owns the results

Money Different models: pay-per-time, pay-per-unit, winner-takes-it-all Define the rewards, analyze trade-offs accuracy vs costs, avoid spam

Love Authors, researchers, Dbpedia community, open access advocates, publishers, casual gamers etc.

Glory Competitions, mentions, awards

7/4/2016 44

Image from Nature.com

What Who

How Why

PRICING ON MTURK

7/4/2016 45

[Ipeirotis, 2008]

ITS NOT ALWAYS JUST ABOUT MONEY

7/4/2016 46

http://www.crowdsourcing.org/editorial/how-to-motivate-the-crowd-infographic/

http://www.oneskyapp.com/blog/tips-to-motivate-participants-of-crowdsourced-translation/

[Kaufmann, Schulze, Viet, 2011]

CURRENT RESEARCH47

CROWDSOURCING RESEARCH

TASK DESIGN TASK ASSIGNMENT QUALITY ASSURANCEINCENTIVES

ENGINEERING

WORKFLOW DESIGN AND EXECUTION

CROWD TRAINING SELF-ORGANIZING CROWDSCOLLABORATIVE

CROWDSOURCING

REAL-TIME DELIVERY EXTENSIONS TO TECHNOLOGIESPROGRAMMING

LANGUAGESAPPLICATION SCENARIOS

IMPROVING PAID MICROTASKS @WWW15Compared effectivity of microtasks on CrowdFlower vs self-developed game Image labelling on ESP data set as gold standard Evaluated

accuracy #labels cost per label avg/max #labels/contributor

For three types of tasks Nano: 1 image Micro: 11 images Small: up to 2000 images

Probabilistic reasoning to personalize furtherance incentives

Findings Gamification and payments work well together Furtherance incentives particularly interesting for top contributors

CROWDSOURCING DBPEDIA CURATION @ISWC13

ENTITY CLASSIFICATION IN DBPEDIA@SEMANTICWEBJOURNAL15Study different ways to crowdsource entity typing using paid microtasks.

Three workflows Free associations Validating the machine Exploring the DBpedia ontology

Findings Shortlists are easy & fast

Popular classes are not enough Alternative ways to explore the taxonomy

Freedom comes with a price Unclassified entities might be unclassifiable Different human data interfaces

Working at the basic level of abstraction achieves greatest precision But when given the freedom to choose, users suggest more specific classes

4.58Mthings

HYBRID NER ON TWITTER @ESWC15

Identified content and crowd factors that impact effectivity #entities in post types of entities content sentiment skipped TP posts avg. time/task UI interaction

Findings Shorter tweets with fewer entities work better Crowd is more familiar with people and places from recent news MISC as a NER category sometimes confusing but useful to identify partial and

implicitly named entities

OBSERVE AND IMPROVE* Elena Simperl, University of Southampton, UK

*with slides by Ramine Tinati, Markus Luczak-Rsch, and others

OVERVIEW

Background to citizen science

Citizen science & human computation

Citizen science & online communities

Citizen science & science

Methods and design guidelines

WHAT IS CITIZEN SCIENCE

EXAMPLE: ZOONIVERSE

EXAMPLE: PYBOSSA

EXAMPLE: VOLUNTEER SCIENCE

EXAMPLE: EYEWIRE

STUDYING CITIZEN SCIENCE: HUMAN COMPUTATION

Task design

Task assignment

Answer validation and aggregation

Contributors performance

Motivation and incentives

STUDYING CITIZEN SCIENCE: ONLINE COMMUNITY

Roles and activities

Patterns of participation

Motivation and incentives

Evolution in time

STUDYING CITIZEN SCIENCE: ESCIENCE

New workflows

New best practices

New publishing models

A FOOTER

When is a citizen science project successful?

LEVELS OF ENGAGEMENT

0

2

4

6

8

10

12

14

16

18

20

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Act

ive

user

s in

%

Month since registration

~1% of participants contribute 72% of Talk & 29% of Task

BEYOND DATA COLLECTION AND ANALYSIS

Discussion and community engagement are integral part of the experience

WORK VS TALK

40.5%

Classifications

Talk

con

trib

utio

ns

Classifications

CHAT AND INSTANT MESSAGING

Microposts

PH SG SW NN GZ CC PF SF AP WS

91%2

0

6

4

10

8

DISCUSSION PROFILES

Deeply engaged volunteers, few

threads but multiple posts within them

9 0.1%

Content producers,

posting across many boards and threads

7 0.1%

Thread followers and PM (one-to-one) talkers

8 0.4%

First to respond and question

answerers 4 1%

Highly active thread starters and answerers across a wide

range of topics

1 2.8%

Infrequent volunteers, single thread posts, no

personal messages

5 5.5%

Watcher and starter of many threads, but not

first to reply3 6.5%

Highly active thread starters

and first to reply back

2 14.6%

Long active volunteers (the

core group), posting

sporadically

6 69.0%

TALKING DOES NOT MEAN LESS WORK

How long do games take to complete depending on when the chat messages are made?

Games took longer to complete when chat messages were being made during the

gaming session, compare to games that had messages before or after.

ECOSYSTEM EFFECTS

Project A

Project B

Project C

Participant X

Part. Y

DESIGNING PLATFORMS

Task specificity

Community development

Task design PR and engagement

Bootstrapping the community

Serendipitous scientific discovery

Engaging with people, supporting profession team

Supporting individuals, finding new scientific discoveries

Obtaining new citizen scientists

Retaining people

Supporting people, improving task completion

Obtaining new citizen scientists

Reinvigorating old users

WHATS NEXT?

Human computation Task assignment: what tasks are interesting/relevant for whom? Peer review, collaborative approaches The role of gamification: is science a game?

Online community Mapping design patterns to online community design frameworks: do CS form a community? Making discussions more effective

Science Citizen science platforms that everyone can use

PUBLICATIONS

R. Tinati, M. Van Kleek, E. Simperl, M. Luczak-Rsch, R. Simpson and N. Shadbolt. Designing for citizen data analysis: a cross-sectional case study of a multi-domain citizen science platform. Proceedings of ACM CHI Conference on Human Factors in Computing Systems 2014 (CHI2015), pages 1-10, 2015.

M. Luczak-Rsch, R. Tinati, E. Simperl, M. Van Kleek, N. Shadbolt and R. Simpson. Why Won't Aliens Talk to Us? Content and Community Dynamics in Online Citizen Science. Proceedings of the International Conference on Weblogs and Social Media (ICWSM2014), 2014.

A FOOTER

E.SIMPERL@SOTON.AC.UK@ESIMPERL

WWW.SOCIAM.ORGWWW.STARS4ALL.EU

7/4/2016 74

mailto:e.simperl@SOTON.AC.UKhttp://www.sociam.org/http://www.stars4all.eu/

FURTHER READING

7/4/2016 75

Social machinestechnology DESIGN and incentivesTutorial overviewTHE ORDER OF SOCIAL MACHINESPEOPLE SUPPLY, CURATE AND ANALYZE DATAOR CREATE CONTENT FROM SCRATCHFOR FREE OR AGAINST A REWARDPEOPLE ARE (ELEMENTARY) PROBLEM SOLVERSSOMETIMES ON ANY GIVEN TOPICPeople like to share their viewsSometimes they work togetherSometimes they competeIt doesnt go well all the timeBut the ones that do well seem to have things in commonSlide Number 14What is a social machine?Slide Number 16SOCIAL MACHINES As crowdsourcingCrowdsourcing:problem solving via open callsDifferent Forms and platforms to CHOOSE FROMFor example: Microtask crowdsourcingSlide Number 21Crowdsourcing & socialITYCrowdSourcing Creative TASKSCrowdSourcing & EthicsMANY QUESTIONS TO ANSWERLunch breakCrowdsourcing designCrowdsourcing design 101Example: ZooniverseDesign your ownIn this talk: CROWDSOURCING data citationSlide Number 32Dimensions of crowdsourcingwhatBroad range of tasksWhoA Large, but Not always diverse crowdSignificant amount of resources and timely deliverySignificant amount of resources and timely delivery (2)HOW: PROCESSHOW: PROCESS (2)How: Validation WHYWhy (2)Pricing on MTurkIts not always JUST about moneyCurrent researchCrowdsourcing Researchimproving paid microtasks @WWW15Crowdsourcing Dbpedia curation @ISWC13Entity classification in DBPedia @SemanticWebJournal15Hybrid NER on Twitter @ESWC15Observe and improve* OverviewWhat is citizen scienceExample: ZooniverseExample: PybossaExample: Volunteer ScienceExample: EyeWireStudying citizen science: human computationStudying citizen science: online communityStudying citizen science: eScienceSlide Number 63Levels of engagementBeyond data collection and analysisWork vs talkChat and instant messagingDiscussion profilesTalking does not mean less workEcosystem effectsDesigning platformsWhats next?Publicationse.simperl@SOTON.AC.UK@esimperlwww.sociam.orgwww.stars4all.eu Further reading