Organic Crowdsourcing Systems

Juho Kim

OrganicCrowdsourcing Systems

with Krzysztof Gajos and Rob Miller

Image courtesy of Christine Daniloff at MIT

Human computationhuman intelligence + computational process

Humans as computational units?

Passive & disengaged crowd

Organic crowdsourcing

Individual & collective benefits to the crowd

Icons by Martha Ormiston from Noun Project

User: personally meaningful activity

Human computation artifact:immediately available in UI

for future users

System:algorithmic

coordination

Organic crowdsourcing system

• Learning via online videos

• Planning a large conference

• Understanding a government budget

Three case studies of organic crowdsourcing





how-to videos online

Navigating how-to videos is hard

find

repeat

skip

Learning with solution structure helps



How to extract & present solution structure at scale?

Subgoal labeling is a costly operation:Requires domain experts and knowledge extraction experts to work together. [Catrambone, 2011]

Insight: Subgoal labeling can be a good learning exercise!

Solution: LEARNERSOURCING

Multi-stage learnersourcing workflow

Learnersourcing Subgoal Labels for How-to Videos. Sarah Weir, Juho Kim, Krzysztof Z. Gajos, & Robert C. Miller.

CSCW 2015.

Sidebar w/ interactive subgoals & steps

Crowdy evaluation

Does participating in learnersourcing improve learning?

Does the learnersourcing workflowproduce good subgoal labels?

Crowdy evaluation

Does participating in learnersourcing improve learning?

Does the learnersourcing workflowproduce good subgoal labels?

Retention test: baseline < Crowdy

Matching expert quality labels

Learnersourcing applications for educational videos

ToolScape [CHI 2014]

Interaction Peaks [L@S 2014] LectureScape [UIST 2014] RIMES [CHI 2015]

Mudslide [CHI 2015]Crowdy [CSCW 2015]





57

420 papers

16 parallel rooms

200 sessions in 15 venues

4 days and 3442 attendees

1pm

3pm

good or bad?

101 102 103 104

Community-guided conference planning

Attendees / Authors / Committee members explore program & express own preferences.

Final schedulewith less conflicts

& more preferences

Scheduling tool encodes community input & recommends

schedule changes.organizer

Committee members

Authors: Customized survey [UIST 2013]

Attendees: Confer

[CHI 2014, HCOMP 2013]

[HCOMP 2014]

Authorsourcing

8,651 preferences and constraints from

645 authors, covering 87% of accepted papers

my paper

fits with my paper does not fit

paper E

paper D

paper B

paper C

paper A

paper Z

Mixed-initiative scheduling tool

Cobi: A Community-Informed Conference Scheduling Tool.Juho Kim, Haoqi Zhang, Paul André, Lydia B. Chilton, Wendy Mackay, Michel Beaudouin-Lafon, Robert C. Miller, Steven P. Dow.

UIST 2013.

Cobi is helping actual conferences





BudgetMap: Issue-driven budget navigation

BudgetMap: Engaging Taxpayers in the Issue-Driven Classification of a Government Budget.Nam Wook Kim, Jonghyuk Jung, Eun-Young Ko, Songyi Han, Chang Won Lee, Juho Kim, Jihee Kim.

CSCW 2016.

Budget tagging interface

Organic crowdsourcing for civic engagement

increased awareness & understanding of

government budget

dynamically generated social issue – budget program links

Organic crowdsourcing design principles

Paid crowdsourcing

simple and concrete task

quality control

microscopic, focused task to workers

cost: money

Organic crowdsourcing

personally meaningful task

incentive design

overall goal & contribution awareness

cost: crowd’s time & effort

Organic Crowdsourcing SystemsJuho Kim | [email protected] | @imjuhokim | juhokim.com

User: personally meaningful activity

Human computation artifact:immediately available in UI

for future users

System:algorithmic

coordination

Engineering

Organic Crowdsourcing Systems