Knowledge Sharing and Crowdsourcing

Xiaolu Yu11/12/2008

Web 2.0

• Online reviews: • Social networking:• Social media sites:• Tagging/folksonomies systems:• Online encyclopendia:• Online blogs: • Online question-answer forums: • CrowdSourcing forums:

• This is the next generation of search. It’s kind of a collective brain of searchable database of everything everyone knows. It is a culture of generosity. The fundamental belief is everyone knows something.

-- Yahoo Research

Knowledge Sharing

• Online encloypendia

• Online question-answer forum

• CrowdSourcing

Question-Answer Forum

• Characteristics– General purpose– Open to public– Large question-answer repository

• Examples– Yahoo Answers– Google Answers– Baidu Knows

Yahoo! Answers• More than 21 million unique users in the U.S. and

90 million worldwide• Largest knowledge-sharing community on the

Web• Anyone can ask and answer questions on any

topic• Connecting people to the information they’re

seeking with those who know it• Providing a way for people to share their

experience and insight

Scoring SystemActionAction PointsPoints

Begin participating on Yahoo! Answers One Time: 100

Ask a question -5

Choose a best answer for your question 3

No Best Answer was selected by voters on your question

Points Returned: 5

Answer a question 2

Deleting an answer -2

Log in to Yahoo! Answers Once daily: 1

Vote for a best answer 1

Vote for No best answer 0

Have your answer selected as the best answer

10

Receive a "thumbs-up" rating on a best answer that you wrote (up to 50 thumbs-up are counted)

1 per "thumbs-up"

Levels

Level Points Questions Answers Comments Stars Ratings Votes

7 25,000+ *unlimited* 100 unlimited 100

610,000 - 24,999

*unlimited* 100 unlimited 100

55,000 - 9,999

*unlimited* 100 unlimited 80

42,500 - 4,999

20 80 40 100 unlimited 80

31,000 - 2,499

15 60 30 100 unlimited 60

2 250 - 999 10 40 20 100 unlimited 40

1 1 - 249 5 20 10 10 0 20

*All limitations are per day

• Every category rates the top ten best answerers based on the number of their best answers (chosen by askers or voters).

• People can either search for questions/answers they are interested in, or actually ask a question

Category Clustering

How has one place you've traveled to changed your life?Travel has had a huge impact on me before, opening my eyes and changing my perspective. Some places I've been have affected me more than others. Where have you been that has changed you and how?

Best Answer - Chosen by AskerI don't think of simply one place changing my perspective as I have traveled the world. What I believe travel does is to afford a most excellent education and to (no pun) broaden horizons. To have the opportunity to see and embrace other peoples' lifestyles and cultures is wonderful.

… …

Wherever we have been, we have found wonderful people, many of whom have become dear friends. These new found friends make travel meaningful. Places may be interesting, but it is the people that make travel worthwhile.

A typical bad answerYes. It made me realize that females are equally stuck-up all over the country.(Still earning 2 points! Noise introduced…)

What do you do with a 15month old girl who holds her breath till she gets what she wants ?

Best Answer - Chosen by AskerYou've gotten a lot of bad answers!

I agree that you should *usually* not give in to your child. What my doctor told me to do when my daughter used to hold her breath (she passed out more than 10 times, including in a store once) was to lay her down so that she would be safe and then leave the room. It is one of the hardest things I have ever done, but it does work.

…

The most difficult thing to understand is that nothing bad will actually happen to your child if she's lying down at the time (if she isn't, she could fall and get hurt). I still find it hard to see a child holding his or her breath, despite knowing this.

Is Earth's solar eclipse unique in the solar system?Can scientific observation and theory prove if are there any other planets in the solar system which have moons that would allow us to observe the exact same effect as our perfect solar eclipse here on Earth? If not, why not?

Best Answer - Chosen by Voters (82% 9 Votes) Not possible on Mercury or Venus as they don't have any moons. Phobos and Diemos, the two Martian moons, are quite tiny (less than 50 kilometres in diameter) and it seems unlikely that solar eclipses could occur on Mars,

…

Jupiter has 63 moons, Saturn 56, Uranus 27, Neptune 13 and Pluto 3 at the latest census. 165 moons around 7 planets in all, Plus there are at least 80 moons around asteroids and minor planets, The asteroid 87 Sylvia has two moons.

Table of Moons by diameter in kilometres…

• How to identify experts?• How to decide if an answer is worthwhile?• What does the quality of questions mean to the

quality of answers and behaviors of users?• Why Google Answers (offering real money

incentive) did not survive?• How would users behaviors change if real

money awards are introduced in to the system?

Network Structure Analysis

Best Answer Selection and Prediction

What are the criteria for best answer selection?• Factual-answer categories – objective• Other categories – subjective

What are the problems with selecting just one best answer?• One best answer per question• Answer-to-question ratio• Noise

What metrics are most predictive of best answers?• Reply length• Number of competing answers• Track record of the user (most significant for technically focused categories)

DiscussionDiscussion

• How to identify experts?• How to decide if an answer is worthwhile?• What does the quality of questions mean to the

quality of answers and behaviors of users?• Is depth sacrificed for breadth?• How to discern good answers that were not

rated as the best answers?• Why Google Answers (offering real money

incentive) did not survive? • Harper et al. (2008) found that the system that offered

money (Google Answers) yielded better answers.• How would users behaviors change if real

money awards are introduced in to the system?

Crowdsourcing

• Crowdsourcing is a neologism for the act of taking a task traditionally performed by an employee or contractor, and outsourcing it to an undefined, generally large group of people, in the form of an open call.

• The term has become popular with business authors and journalists as shorthand for the trend of leveraging the mass collaboration enabled by Web 2.0 technologies to achieve business goals.

Crowd Sourcing Examples and differences

• Taskcn • eLance• TopCoder• Google Answers

Crowd Sourcing Examples and differences

• eLance and TopCoder: Tasks are attempted after the requester chooses a provider or a team of providers based on his/their credentials and proposals.

• Google Answers: participations are limited to expert answerers recruited; answerers would have exclusive locks on the tasks for a period of time.

• Taskcn: users submit their work directly (new Taskcn offers multiple approaches based on the difficulty and complexity of tasks. )

Taskcn.com• 1.7 million registered users • In less than 2 years, requested solutions for nearly 3100 tasks and

543,000 solutions proposed • A user offers a monetary award for a question or task and other

users provide solutions to compete for the award.• The website plays the role of the third party by collecting the money

from the requester and distributing the award to the winner(s) who is (are) decided by the requester.

• The website takes a small portion of the award as a service fee.• Socially stable: a core group of users who repeatedly propose and

win• Incentivized mechanism: Potential monetary award encourages

people’s participation.

Users are Learning over Time

• Submit later• Choose less popular tasks• Choose Tasks with higher winning odds• Raise award expectation

• However, average users fail to improve.

Submit Later

• Winners consistently submit later than non-winners over time, implying greater effort.

Combination of Multiple Strategies by Serious Users

• Higher winning probability– Less popular tasks

• Higher expected award• Tasks requiring greater skill• Avoid tasks of higher workload

– Workload is negatively correlated with award.

Rich Getting Richer?

• On average, users do not increase (decrease) their chances of winning although they implement strategies.

• In contrast, a very small core of successful users manage to win multiple tasks as well as to increase their win-to-submission ratio over time.

• Reputation enhancement or learning process?

Incentive, task complexity & user strategy

• Why incentive would change users participation and strategy?– It encourages users to contribute expertise beyond

simple question answering, as might occur on no-fee sites such as Yahoo! Answers.

– The task submitted must be necessarily of relatively low complexity and effort since given no guarantee of awards.

– (New Taskcn feature): In the case that task is attempted after the requester makes his choice, the tasks are relatively more complex.

Winning Strategy

• Target group: a group of winners who get more efficient at winning over time.

Observation: quickening in the succession of wins

Winning Strategy

• Successful in selecting tasks of lower popularity than those chosen by average users from their first attemps.

Winning Strategy

• Successfully selecting less popular tasks from the very first attempts

• Always submitting later than others

• Winners are better at starting and sticking with such strategies that will improve their chances of winning.

Core Group Contribution• A large fraction of winning task solutions is

contributed by a small core group of individuals.• Such skewness of contribution to Internet peer

production systems have been widely observed.– Wikipedia

• Participation is open to anyone, but a large portion of the content is contributed by a small minority of the participants.

Design Implications

• Identify the core group of winners early• Incentivize the core group (if a core group is

necessary)• Drive large number of prospective users towards a

site• Make task awards and skills and level required by

the tasks commensurate