Upload
henry-copeland
View
216
Download
4
Embed Size (px)
Citation preview
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
Easy (and cheap) data
Iperotis 2010b
Median hit - $006
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
gt500000 ldquoWorkersrdquo
~35-40 US 30-50 India+ ~100 countries
Heinrich Heine Norenzayan 2010
Western
Educated
Rich
Industrialized
Democracies
ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
N=126(135 tested)$270 + 10
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
Easy (and cheap) data
Iperotis 2010b
Median hit - $006
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
gt500000 ldquoWorkersrdquo
~35-40 US 30-50 India+ ~100 countries
Heinrich Heine Norenzayan 2010
Western
Educated
Rich
Industrialized
Democracies
ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
N=126(135 tested)$270 + 10
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Easy (and cheap) data
Iperotis 2010b
Median hit - $006
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
gt500000 ldquoWorkersrdquo
~35-40 US 30-50 India+ ~100 countries
Heinrich Heine Norenzayan 2010
Western
Educated
Rich
Industrialized
Democracies
ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
N=126(135 tested)$270 + 10
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
gt500000 ldquoWorkersrdquo
~35-40 US 30-50 India+ ~100 countries
Heinrich Heine Norenzayan 2010
Western
Educated
Rich
Industrialized
Democracies
ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
N=126(135 tested)$270 + 10
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
gt500000 ldquoWorkersrdquo
~35-40 US 30-50 India+ ~100 countries
Heinrich Heine Norenzayan 2010
Western
Educated
Rich
Industrialized
Democracies
ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
N=126(135 tested)$270 + 10
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Heinrich Heine Norenzayan 2010
Western
Educated
Rich
Industrialized
Democracies
ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
N=126(135 tested)$270 + 10
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Easy (and cheap) data
Diverse
Intuition pump
(My) reasons for using crowdsourcing
N=126(135 tested)$270 + 10
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
N=126(135 tested)$270 + 10
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Dale amp Lupyan 2011
How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc
r = 2
3 p
= 0
25Lo
git r
eg o
dds
1048576 ra
tio =
78
p =
028
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)
55 Female 45 Malendash Similar to other internet panels (eg
Goldstein)
Age ndash Mean 30 yrs ndash Median 32 yrs
Mean Income $30000 yr
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Whoworks
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India
ldquoMTurk money is irrelevantrdquondash 12 US 10 India
ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India
Ross et al rsquo10 Ipeirotis rsquo10
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Why
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing
Wha
t
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers
IntermediariesCrowdFlowerSmartsheetcom
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings
hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Whorequests
Iperotis 2010b
Wha
t
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Anatomy of a HIT
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)
HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT
Not everyone is eligible to do every HIT
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Some HIT groups have many HITs
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoBlackrdquo
bullbullbull
Assignment 2 ldquoNightrdquo
Assignment 3 ldquoBlackrdquo
Alice
Bob
Charlie
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Which is the better translation for Taacutey o Blacko Night
HIT 1
Which is the better translation for Nedj o Cleano White
HIT 2
HIT GROUP
Assignment 1 ldquoWhiterdquo
bullbullbull
Assignment 2 ldquoWhiterdquo
Assignment 3 ldquoWhiterdquo
Alice
Bob
DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Build HIT
Test HIT
Post HIT
Reject or Approve HIT
Search for HITs
Accept HIT
Do work
Submit HIT
Requester Worker
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Pay rate can affect quantity of workPay rate does not have a big impact on quality
Pay per Task Pay per Task
Num
ber o
f Tas
ks C
ompl
eted
Accu
racy
How much $
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Completion Time
3 6-question multiple choice surveys
Launched same time of day day of week$001 $003 $005
Past a threshold pay rate does not increase speed
Start with low pay rate work up
Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Turker Community
Reputation of Workers is given by approval rating
Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Validity
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
How
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Qualtrics and Mturk
-As good as point and click gets
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Number of choices
Att
rib
ute
Friendly
Safe
Smooth
Rounded
Round
Short
Fat
Small
Wide
Large
Tall
Thin
Sharp
Narrow
Spikey
Dangerous
Pointy
Unfriendly
Angular
0 5 10 15 20
Labelcrelchfoove
Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like
Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
If you can code it you can run it
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Test run 1
bull Recruited through social network contactsbull Overall about ~60 players produced 1377
squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Test Run 2
bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk
bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate
ndash $015 hour
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
TipsBe creative Donrsquot try to force a lab study into a Turk format
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
Thibodeau amp Boroditsky 2011
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
TipsValidate the work when necessary
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
TipsTake the workers perspective
Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results
Ask workers for feedback
Be aware of your reputationTurkopticon Turker Nation
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
(1) Data through intervention or interaction with the individual or(2) Identifiable private information
Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes
Interaction includes communication or interpersonal contact between investigator and subject
Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects
IRB