Methods and techniques (2/6)

Methods and techniques to analyze and design incentivized

semantic applications

www.insemtives.eu

Roberta Cuel, University of Trento, IT and Markus Rohde, University of Siegen, DE

ISWC 2010

www.insemtives.eu04/14/2023 2

Why is my app not as successful as Facebook…

04/14/2023 www.insemtives.eu 2

• Motivation and incentives– Reciprocity – Reputation – Competition– Altruism– Self-esteem – Fun– Money

• This session is about the methods and techniques you can use – To study your scenario prior to

application design– To evaluate your incentives strategy

and adjust your application

www.insemtives.eu

Overview

• Basic notions and definitions• Methods and techniques for analysis, design

and evaluation• Case studies

04/14/2023 3

www.insemtives.eu

BASIC NOTIONS AND DEFINITIONS

Basic notions and definitionsMethods and techniques for analysis, design and evaluation

Case studies

04/14/2023 4

www.insemtives.eu

Starting with motivation…

• Basic tenets of organizational behavior – Performance : f(ability*motivation)

04/14/2023 5

www.insemtives.eu

Motivation and incentives

• Intrinsic vs extrinsic motivations• Incentives are ‘rewards’ assigned to performer

to make him/her commit– Can be totally uncorrelated to the nature of the

task – Need to be compliant with the values and beliefs

of the acting person

04/14/2023 604/14/2023 www.insemtives.eu 6

www.insemtives.eu

Example: DBPedia

The DBpedia community collaborates using the following tools– Mailing lists, Facebook group– Blog, quality assurance and bug tracking– Development

• “The framework is easily extended and we offer the possibility to do this yourself, increasing the likelihood of your desired data being included in the next DBpedia release.”

– DBpedia consulting (money)

04/14/2023 7


Typology of motivationsMotivations Internal

(embedded in structure, e.g., task, tools)

External(additional to structure, external re-inforcements)

Intrinsic(predispositioned in person, e.g., drives, needs, desires )

Fun, joy, gaming,interest, satisfaction, self-actualization, self-re-inforcement

Social appreciation, reputation, love, trust, social capital, community support

Extrinsic(additional to personal predispositions, extern re-inforcements )

Usability, sociability,Design-for-fun, curiosity,community-building support

Material/financial capital, money, rewards, prices, medals, credit points

Structure

Person

Example: FLOSS software (Ghosh & Prakash, in Lerner & Tirole, 2005)

www.insemtives.eu

METHODS AND TECHNIQUES FOR ANALYSIS, DESIGN AND EVALUATION


Case studies

04/14/2023 9

www.insemtives.eu

Overview of methods and techniques

• To analyze your scenario and design your application– Game theory, mechanism design– Participatory design and end-user development

• To evaluate an incentives strategy in its context– Interviews and questionnaires and field experiments – Heuristic evaluation, guideline reviews– Cognitive and pluralistic walk-throughs– Participatory/User evaluation


www.insemtives.eu

Game theory

• Game theory is a formal way to analyze interaction among a number of rational agents who behave strategically– The rational agents: players involved in the situation (best choice)– A number of players: more than one– Rationality/payoffs: what are the players’ preferences over the

outcomes of the game– The interactions: one player’s behavior affects another – The rules: who moves when, what do they know, what can they do– The outcomes: what is the outcome of the game (for each move)

04/14/2023 11

www.insemtives.eu

Mechanism design

• Mechanism design is about how to translate game theory in effective behavior– To design rules such that a desired set of outcomes

happens– Alignment of interests between parties and production of

maximum social welfare

• Relevant variables– Structural and personal motivations– Goal and nature of good produced– Tasks and skills required– Social structure

04/14/2023 12

www.insemtives.eu

Analysis matrix

• A multidimensional tool that allows for an analysis of a particular task with respect to the variables relevant for mechanism design

04/14/2023 13

Goal Tasks Social structure Nature of good produced

Skill variety/level

Communication level

Variety of Hierarchy-neutral

Private good Skilled ability

Participation level

Specificity of Hierarchical Public good Competence

Clearness level Identification with

Common resourceClub good


The matrix in practice: semantic annotation

• Goal of the annotation or ontology population exercise• Task, or more typically, an ordered collection of tasks into

which the annotation exercise can be broken down• Social structure, a stylized and simplified set of social

relationships among the subjects participating in the exercise• Nature of good, a stylized description, in game-theoretical

terms, of the relationship between what good is produced and who consumes it

• Required skills of the agents to complete the annotation task


www.insemtives.eu

The matrix in practice: crowd-sourcing of ontology evolution

• Goal: communication and participation level– Coordination (free-open)

• Interests are already aligned (free riding)• Private and public benefit coincide

• Task: variety vs specificity • Social structure: hierarchy neutral (large vs small

groups)• Nature of good: public good non-rival (use does not

limit use of others), non-exclusive (open access)• Skill variety/level: Skilled ability


www.insemtives.eu

Participatory Design (PD)

• To develop software for the participation of end-users, we propose a participatory way of designing these software tools, integrating potential users by participatory design methods

• Participatory Design– Improvement of the participation of workers in software development

processes– The cooperation between software developers and end-users– Participate in IT development projects as experts of their own work

04/14/2023 16


Methods for PD

• Participatory-design projects combine– Design-by-doing methods – PD workshops– Scenarios – Different forms of prototyping (mockups, rapid

prototypes)– Work organization games – Ethnographic methods


End-User Development (EUD)

• EUD is about designing highly flexible systems that enable users to participate during the use of the system by adapting and modifying the tools according to their needs/preferences

• Goal: Empower end-users to develop and adapt systems themselves by designing these systems to be easy to– Understand– Learn– Use– Teach


How to design an incentivized applicationIdeally: field desk lab field

Analyze the domain and find yourselves in the matrixes Find the relevant point of that situation (goal and tasks)Focus on a small group of individuals (social structure)Analyze their motivation (internal/external intrinsic/extrinsic)Analyze the other relevant variables (nature of good being produced, kill variety/level)

Design a simplest possible model that can effectively support contributorsTest and get feedback Fine-tune the experiment and add other elements


Methods

• Domain observations (second-hand data)• Ethnography or qualitative face-to-face interviews • Questionnaires• Observations with selected individuals • Quantitative analysis (data collections)• Usability-design methods


www.insemtives.eu

CASE STUDIES


Case studies

04/14/2023 21

www.insemtives.eu

Examples from our project

• Telefónica Investigación y Desarrollo (TID - Spain): Knowledge Intranet Platform

• Pepper’s Ghost (PGP-UK): Virtual Games/Worlds• Seekda! (Austria): Web Services Search Portal

04/14/2023 22


Field work: TID

• Domain analysis– Site visit, semi-structured, qualitative interviews

• Communication processes• Existing usage practices and problems• Existing tools/solutions• Semantic annotation solutions

– Tape recording, transcription– Data analysis per ex-post categorization

• Focus group discussion– Usability lab tests– Expert walkthroughs


Field work: TID (2)


• Find TID in the matrix– Goal: communication and participation (contribute to the

knowledge portal) – Social structure (various structures co-exist)

• Strongly hierarchical organization (control is an issue) • working groups and community of experts

– Nature of good: public good vs. private, club goods– Skill variety/level: Skilled ability (knowledge workers)– Motivations: fun, visibility, reputation, promotion, money


Field work: PGP• Domain analysis– Data analysis (Benchmark: Galaxyzoo, Moonzoo)– Usability lab tests and expert walkthroughs– Collect data on users such as logs files

• Find PGP in the matrix– Goal: participation level– Task: specific– Social structure: hierarchy neutral– Nature of good: public good non-rival (use does not limit

use of others), non-exclusive (open access)– Skill variety/level: non skilled ability


Field work: SEEKDA

• Domain analysis– Interviews and workshops – Replicating Telefonica model

• Find SEEKDA in the matrix:– Goal: communication and participation level– Task: variety vs. specificity – Social structure: hierarchy neutral (mainly)– Nature of good: public good non-rival (use does not limit

use of others), exclusive, or club good– Skill variety/level: Skilled ability

www.insemtives.eu

Thank you

Any question?

04/14/2023 27

Documents

Methods and techniques (2/6)