Upload
jeffry-watts
View
215
Download
0
Embed Size (px)
Citation preview
Task and Workflow Designin Human Computation
KSE 652 Social Computing System Design and Analysis
Uichin Lee
TurKit: Human Computation Algorithms on Mechanical Turk
Greg Little, Lydia B. Chilton, Rob Miller, and Max Goldman
(MIT CSAIL)UIST 2010
Workflow in M-Turk
HIT
HIT
HIT
HIT
HIT
HIT
Data Collected
in CSV File
Requester posts HIT Groups to
Mechanical Turk
Data Exported for Use
Workflow: Pros & Cons
• Easy to run simple, parallelized tasks.• Not so easy to run tasks in which turkers
improve on or validate each other’s work.
• TurKit to the rescue!
The TurKit Toolkit
• Arrows indicate the flow of information.
• Programmer writes 2 sets of source code:– HTML files for web
servers– JavaScript executed by
TurKit
• Output is retrieved via a JavaScript database.
Turkers
Mechanical Turk
Web Server TurKit
*.html *.js
Programmer
JavaScript Database
Crash-and-rerun programming model
• Observation: local computation is cheap, but the external class costs money
• Managing states over a long running program is challenging– Examples: Computer restarts? Errors?
• Solution: store states in the database (just in case)• If an error happens, just crash the program and re-run by
following the history in DB– Throw a “crash” exception; the script is automatically re-run.
• New keyword “once”: – Remove non-determinism– Don’t need to re-execute an expensive operation (when re-run)
• But why should we re-run???
Example: quicksort
Parallelism
• First time the script runs, HITs A and C will be created
• For a given forked branch, if a task fails (e.g., HIT A), TurKit crashes the forked branch (and re-run)
• Synchronization w/ join()
MTurk Functions
• Prompt(message, # of people)– mturk.prompt("What is your favorite color?", 100)
• Voting(message, options)• Sort(message, items)
VOTE() SORT()
TurKit: Implementation
• TurKit: Java using Rhino to interpret JavaScript code, and E4X2 to handle XML results from MTurk
• IDE: Google App Engine3 (GAE)
Online IDE
Exploring Iterative and Parallel Human Computation Processes
Greg Little, Lydia B. ChiltonMax Goldman, Robert C. Miller
HCOMP 2010
HC Task Model
• Dimension: – Dependent (iterative) or independent (parallel) tasks – Creation and decision tasks
• Task model examples
Creation tasks (creating new content): e.g., writing ideas,
imagery solutions, etc.
Decision tasks (voting/rating): e.g., rating quality of a description of an
image
HC Task Model
• Combining tasks: iterative and parallel tasks
Iterative pattern: a sequence of creation tasks where the result of each task feeds into the next one, followed by a comparison task
Parallel pattern: a set of creation tasks executed in parallel, followed by a task of choosing the best
Experiment: Writing Image Description
• Iterative vs. parallel; each 6 creation tasks ($0.02), followed by rating tasks (1-10 scale, $0.01)
Experiment: Writing Image Description
• Turkers in iterative condition gave better description while parallel condition always shows an empty text area.
Experiment: Writing Image Description
• Average rating after n iterations– After six iterations: 7.9 vs. 7.4, t-test T29=2.1, p=0.04
iterative
parallel
Experiment: Writing Image Description
• Length vs. rating: positive correlation
• The two outliers (circled) represent instances of text copied from the Internet (with superficial description)
Length (characters)
Ratin
g
Experiment: Writing Image Description
• Work Quality:– 31% mainly append content at the end, and make only minor
modifications (if any) to existing content; – 27% modify/expand existing content, but it is evident that they use
the provided description as a basis;– 17% seem to ignore the provided description entirely and start over;– 13% mostly trim or remove content; – 11% make very small changes (adding a word, fixing a misspelling,
etc);– 1% copy-paste superficially related content found on the internet.
• Creating vs. improving (takes about the same time, avg. 211 seconds)
Experiment: Brainstorming
Experiment: Brainstorming
• Iterative work: higher average rating– Biased thinking: e.g., tech -> xxtech -> yytech
• Parallel work: diversity, higher deviation (rating) – No iteration for brainstorming
Iteration Rating
Avg.
Rati
ng
iterative
parallel
Example: Blurry Text Recognition
Example: Blurry Text Recognition
• Iterative performs better than parallel
Iteration
Accu
racy
Summary
• TurKit: a flexible programming tool for m-turk
• Various work-flow can be designed; e.g., iterative, parallel, and hybrid
• Iterative performs better than parallel in several cases (e.g., image description, brainstorming, text recognition)
Turkalytics: Real-time Analytics for Human Computation
Paul Heymann and Hector Garcia-MolinaWWW'11
Basic Buyer human programming• A human program generates forms; advertised through a marketplace. • Workers look at posts, and then complete the forms for compensation.
Game Maker human programming• The programmer writes a human program and a game. • The game implements features to make it fun and difficult to cheat. • The human program loads and dumps data from the game.
Human Processing programming
Human Processing programming• Task description:
– Input, output, web forms, human driver, other information– Human task instance
• Human drivers: interact with workers– Functions: initialization (forms, games), retrieving results – “Human Program” accesses workers via “human drivers”
• Recruiters: post task instances into the marketplaces, (by working with marketplace drivers)– Marketplace driver provides an interface to marketplaces
(description) (instance)
Turkalytics
• Challenge: collecting reliable data about the workers and the tasks they perform
• Why?– If a task is not being completed, is it because no workers
are seeing it? Is it because the task is currently being offered at too low a price?
– How does the task completion time break down? – Do workers spend more time previewing tasks or doing
them? – Do they take long breaks? – Which are the more “reliable” workers?
Interaction Model
• Search-Preview-Accept (SPA) model
Interaction Model• Search-Continue-RapidAccept-Accept-Preview (SCRAP)
Continue completing a task that was accepted but not submitted
Accept the next task in a HITGroup w/o previewing it
Turkalytics Data Models
Turkalytics ArchitectureClient-side javascript: ta.js Log Server
Client-side javascript: ta.js
ta.js
ta.js
Ajax: POST
Log messages (JSON )
Analysis Server
Log messages (JSON )
Implementation: client-side Javascript
• Requester embeds a Turkalytics script (ta.js) into a HIT (when designing a HIT)– Monitoring: Detect relevant worker data and actions.– Sending: Log events by making image requests to the
log server (ajax: POST)
Implementation: ta.js -- client-side JavaScript
• ta.js’s monitoring activities:– Client Information: Worker’s screen resolution? What
plugins are supported? Can ta.js set cookies?– DOM Events: Over the course of a page view, the
browser emits various events (e.g., load, submit, before unload, and unload events)
– Activity: listens on a second-by-second basis for the mousemove, scroll and keydown events to determine if the worker is active or inactive.
– Form Contents: examines forms on the page and their contents; logs initial form contents, incremental updates, and final state.
Implementation: log/analysis
• Log Server:– Simple web app built on Google’s App Engine. – Receives logging events from clients running ta.js and saves them
to a data store. • IP address, user agent, and referer, etc
• Analysis Server: – Periodically polls the log server to download any new events that
have been received – Event inserted into DB, considering the following:
• Time constraints: data availability to analysis server• Dependencies: if events are dependent on one another• Incomplete input: if all events are not received yet..• Unknown input: what if unexpected input is received?
Implementation: analysis
// what type of data (event) is sent // actual data for a given type
Detailed info about task
// session ID
Experiments• Tasks:
– Named Entity Recognition (NER): This task, posted in groups of 200 by a researcher in Natural Language Processing, asks workers to label words in a Wikipedia article if they correspond to people, organizations, locations, or demonyms. (2, 000 HITs, 1 HIT Type, more than 500 workers.)
– Turker Count (TC): This task, posted once a week by a professor of business at U.C. Berkeley, asks workers to push a button, and is designed just to gauge how many workers are present in the marketplace. (2 HITs, 1 HIT Type, more than 1, 000 workers each.)
– Create Diagram (CD): This task, posted by the authors, asked workers to draw diagrams for this paper based on hand drawn sketches
Experiments: origin of workers
• GeoLite City DB from MaxMind to geolocate all remote users by IP address
Experiments: worker characteristics
Experiments: states/actions
• RapidAccept is quite popular (Continue is rare)
Experiments: # previews• Artificial recency for NER/CD (keep making them near the top in the list):
NER and CD exhibit less severe drop as opposed to TC
ArtificialRecency
Experiments: activity vs. delay
• Average active and total seconds for each worker who completed the NER task (correlation 0.88)
Discussion
• Multi-tasking users? Activity vs. working time• Privacy??– We can collect as much as we can..– How about Google Analytics? Any web pages that we visit
can collect such information…
• False data injection?• How can we better utilize the dataset?– Re-designing existing tasks, pricing, etc. (or mining user
behavior?)