Engineering human-based services in
complex processes
Hong-Linh Truong
Distributed Systems Group,
Vienna University of Technology
[email protected]://dsg.tuwien.ac.at/staff/truong
1ASE Summer 2014
Advanced Services Engineering,
Summer 2014 – Lecture 9
Advanced Services Engineering,
Summer 2014 – Lecture 9
Outline
Human capabilities for complex processes
Human service units
Incorporating human units into complex services
Frameworks
Some issues and solutions
ASE Summer 2014 2
Human-based services for solving
complex problems (1)
3
We need to actively finding and binding human
capabilities into the application!
We need to actively finding and binding human
capabilities into the application!
Humans and software for
solving complex problems
ASE Summer 2014
Human-based services for solving
complex problems (2)
4
But how to program human-based services and
software-based services together?
But how to program human-based services and
software-based services together?
ASE Summer 2014
Example: some common tasks in
data analytics
ASE Summer 2014 5
Human activities in Data Analytics
LabelingAnnotating
CleansingEvaluating
data/contentDectecting patterns
Classifying objects
Steering analytics
Human service units in data
analytics -- functions
Evaluating: is the quality of picture good?
Classifying: is it a man‘s or a woman‘s picture?
Decting: any unidentified object in a picture?
Labeling: adding location information of a picture
Cleansing: remove duplicated pictures
Steering: the quality of picture is bad, should we
continue to merge it with others?
ASE Summer 2014 6
How to model such functions for human units ?
E.g., with WSDL or REST?
How to model such functions for human units ?
E.g., with WSDL or REST?
Human service units
ASE Summer 2014 8
Human service unit
Human service unit
Functions
Non-functional
parameters
Service models
Technical interfaces
Provisioning mechanisms
Interaction models
Human acting as a „service unit“Human acting as a „service unit“
Human service units
9
Modeling type of
units (e.g.,
computation, data,
monitor,) and their
dependencies
Consumption,
ownership, provisioning,
price, etc.
ElasticService
Unit
Service model
Unit Dependency
ElasticCapability
Function
The functional
capability of the unit
and interface to
access the function
Capabilities to be elastic
under different
requirements
SoftwareSoftware PeoplePeople
VolunteersProfessionals
ThingThing
Resources
ASE Summer 2014
Forms of human service
Individual Compute Unit
An individual is treated like „a processor“ or “functional
unit“. A service can wrap human capabilities to support
the communication and coordination of tasks
Social Compute Unit
A set of people and software that are initiated and
provisioned as a service for solving tasks
Web services interfaces can be built
Different pricing models and different
quality models
ASE Summer 2014 10
Human service units – provisioning
mechanisms (1)
An infrastructure can be introduced for
accessing many ICUs in a crowd
Allow people to register their service unit capabilities
Facilitate communication, task bidding, retrieval and
result delivery
Act like a marketplace: multiple providers and multiple
consumers
ASE Summer 2014 11
ICU ICU
ICUICU
ICUICU ICU
CrowdCrowd
Human service units – provisioning
mechanisms (2)
An „infrastructure-as-a-service“ for ICUs
Facilitate communication, task retrieval and result
delivery
Single ICUaaS provider and multiple consumers
ASE Summer 2014 12
ICU ICU
ICUICU
ICUICU ICU
Cloud
provisioning
models ICUICU
Human service units – provisioning
mechanisms (3)
An „infrastructure-as-a-service“ for SCUs
Facilitate communication, task retrieval and result
delivery
Single SCUaaS provider and multiple consumers
ASE Summer 2014 13
SCUSCU
SCUSCU
SCUSCU
SCU
Cloud
provisioning
models SCUSCU
Human service units – technical
interfaces (1)
ASE Summer 2014 14
People
Web Servicce
(REST/SOAP)
People
Web page
Analytics
Activity
Analytics
Activity
Human service units – technical
interfaces (2)
ASE Summer 2014 15
Email/SMS/Instant
Messaging
People
Web page
Web Servicce
(REST/SOAP)Communication
Services
People
Analytics
ActivityAnalytics
Activity
Human service units – interaction
model
ASE Summer 2014 16
ICU/SCU
Task queue
Scheduler
ICU/SCU
Task queue
Scheduler
Analytics
Activity
Analytics
Activity
ICU/SCU
Analytics
Activity
Human service units -- NfPs
ASE Summer 2014 17
Which are important
considerations when
interpreting NfP for human
units?
Which are important
considerations when
interpreting NfP for human
units?
Incorporating human units into
complex processes
How to provision and employ human compute
units?
How to select human units?
Where to place human units in data analytics
and why?
How to monitor and test human units in data
analytics?
ASE Summer 2014 18
Qurk system architecture (1)
ASE Summer 2014 20
Source: Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller. 2011. Human-powered sorts and joins. Proc. VLDB
Endow. 5, 1 (September 2011), 13-24.
Source: Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller. 2011. Human-powered sorts and joins. Proc. VLDB
Endow. 5, 1 (September 2011), 13-24.
Qurk system architecture (2)
ASE Summer 2014 21
Source: Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller. 2011. Human-powered sorts and joins. Proc. VLDB
Endow. 5, 1 (September 2011), 13-24.
Source: Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller. 2011. Human-powered sorts and joins. Proc. VLDB
Endow. 5, 1 (September 2011), 13-24.
Jabberwocky approach (1)
ASE Summer 2014 22
Source: Salman Ahmad, Alexis Battle, Zahan Malkani, Sepandar D. Kamvar: The jabberwocky programming environment for structured
social computing. UIST 2011: 53-64
Source: Salman Ahmad, Alexis Battle, Zahan Malkani, Sepandar D. Kamvar: The jabberwocky programming environment for structured
social computing. UIST 2011: 53-64
Jabberwocky approach (2)
ASE Summer 2014 23
Source: Salman Ahmad, Alexis Battle, Zahan Malkani, Sepandar D. Kamvar: The jabberwocky programming environment for structured
social computing. UIST 2011: 53-64
Source: Salman Ahmad, Alexis Battle, Zahan Malkani, Sepandar D. Kamvar: The jabberwocky programming environment for structured
social computing. UIST 2011: 53-64
Automan approach
ASE Summer 2014 24
Source: Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, Andrew McGregor: AutoMan: a platform for integrating human-based
and digital computation. OOPSLA 2012: 639-654
Source: Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, Andrew McGregor: AutoMan: a platform for integrating human-based
and digital computation. OOPSLA 2012: 639-654
SW4H approach (1)
ASE Summer 2014 25
Karastoyanova, Dimka; Dentsas, Dimitrios; Schumm, David; Sonntag, Mirko; Sun, Lina; Vukojevic, Karolina: Service-based Integration of
Human Users in Workflow-driven Scientific Experiments. In: Proceedings of the 8th IEEE International Conference on eScience (eScience
2012
Karastoyanova, Dimka; Dentsas, Dimitrios; Schumm, David; Sonntag, Mirko; Sun, Lina; Vukojevic, Karolina: Service-based Integration of
Human Users in Workflow-driven Scientific Experiments. In: Proceedings of the 8th IEEE International Conference on eScience (eScience
2012
SW4H approach (2)
Similar concepts in
collaborative working
environments but
integrated into
workflows
Do not discuss about
where and how to
select human units
ASE Summer 2014 26
Karastoyanova, Dimka; Dentsas, Dimitrios; Schumm,
David; Sonntag, Mirko; Sun, Lina; Vukojevic, Karolina:
Service-based Integration of Human Users in Workflow-
driven Scientific Experiments. In: Proceedings of the 8th
IEEE International Conference on eScience (eScience
2012
Karastoyanova, Dimka; Dentsas, Dimitrios; Schumm,
David; Sonntag, Mirko; Sun, Lina; Vukojevic, Karolina:
Service-based Integration of Human Users in Workflow-
driven Scientific Experiments. In: Proceedings of the 8th
IEEE International Conference on eScience (eScience
2012
VieCOM -- incorporate humans into a
programming paradigm (1)
• Abstracting human compute units as program elements
• Extending programming languages to support human compute units
• Data/control flows via extensible APIs
Programming languages
• Shared memory (e.g., human –software – human), message passing (human-to-human), artifact-centric, etc., via APIs working atop the compute unit abstraction layer
Multiple programming
models
• Computing capability /profile management: human computing power, reputation and incentive models
• Monitoring and enforcing incentives/rewards, quality of results, availability
• Communication between human-middleware, among Individual Compute Units (SCU)/Social Compute Units (SCU) for exchanging artifacts and comprehensing l tasks
Execution environment
28 http://dsg.tuwien.ac.at/research/viecomASE Summer 2014
VieCOM-- incorporate humans into a
programming paradigm (2)
VolunteersProfessionalsTeamIndividual
Service-based Middleware
CommunicationCommunicationMonitoringMonitoringCapability/Profile
Management
Capability/Profile
Management
Provisioning/Negotiation/Execution APIProvisioning/Negotiation/Execution API
Abstraction of Human-based Compute Units
SCUSC
U
SC
U SCU SCU
Program languages and programming models
Program
elements
Software
Compute
Units
program human actions
and dependencies
program incentive condition
and rewarding action
program result evaluation
method
Human-to-middleware
interfaces:
•visualization of collective tasks
•embedding of common forms
•mobile app
29ASE Summer 2014
Cloud of hybrid service units
Cloud of HBS: A cloud of HBS includes HBS that can be provisioned, deployed, and utilized on-demand based on different pricing and incentive models.
Cloud of HBS: A cloud of HBS includes HBS that can be provisioned, deployed, and utilized on-demand based on different pricing and incentive models.
Cloud of hybrid services: A cloud of hybrid services includes SBS and HBS that can be provisioned, deployed and utilized on-demand based on different pricing andincentive models.
Cloud of hybrid services: A cloud of hybrid services includes SBS and HBS that can be provisioned, deployed and utilized on-demand based on different pricing andincentive models.
Sevilla
SCU for independent tasks
Application /
Workflow
SCU
Provisoning Middleware
Human-Task
Requests
ICU Clouds● Task-based crowdsourcing platforms
● Collections of experts on SN
● Enterprise ICU pools
Consumer
Problems
- Complex tasks
- Quality control
- Flexible quality requirements
Problems
- Complex tasks
- Quality control
- Flexible quality requirements
SCU
ICU
Human-Task
Requests
- ICU: individuals
- SCU: a collective of
collaborative individuals
- ICU: individuals
- SCU: a collective of
collaborative individuals
Elastic SCU provisioning atop ICUs
Elastic profileSCU (pre-)runtime/static formation
Cloud APIs
Muhammad Z.C. Candra, Hong-Linh Truong, and Schahram
Dustdar, Provisioning Quality-aware Social Compute Units in
the Cloud, ICSOC 2013.
Muhammad Z.C. Candra, Hong-Linh Truong, and Schahram
Dustdar, Provisioning Quality-aware Social Compute Units in
the Cloud, ICSOC 2013.
Algorithms
Ant Colony
Optimization
variants
FCFS
Greedy
SCU
extension/reduction
Task reassignment
based on trust, cost,
availability
Mirela Riveni, Hong-Linh Truong, and Schahram
Dustdar, On the Elasticity of Social Compute Units,
CAISE 2014
Mirela Riveni, Hong-Linh Truong, and Schahram
Dustdar, On the Elasticity of Social Compute Units,
CAISE 2014
Selecting human units
Do not select at all
Let human units bid the tasks
E.g., in crowdsourcing platforms
Static/fix mapping
E.g., using static information for human-task mapping
Simple selection techniques
Using the requirement of the task to find the suitable
human units based on their capabilities
Complex selection techniques
Utilizing complex dependency graphs to find suitable
human units
ASE Summer 2014 33
Selecting SCU based on task
graphs
34
SCU Formation
Algorithms
Business As
Usual
Corrective
Action
HBS
Constraints
Hong Linh Truong, Schahram Dustdar, Kamal Bhattacharya: Programming Hybrid Services in the Cloud. ICSOC 2012: 96-110Hong Linh Truong, Schahram Dustdar, Kamal Bhattacharya: Programming Hybrid Services in the Cloud. ICSOC 2012: 96-110
ASE Summer 2014
Utilizing hybrid services for
evolving/dependent task graphs
Hong-Linh Truong, Schahram
Dustdar, Kamal Bhattacharya
"Programming Hybrid Services
in the Cloud", 10th International
Conference on Service-oriented
Computing (ICSOC 2012),
November 12-16, 2012, Shanghai,
China. Best Paper Award.
Hong-Linh Truong, Schahram
Dustdar, Kamal Bhattacharya
"Programming Hybrid Services
in the Cloud", 10th International
Conference on Service-oriented
Computing (ICSOC 2012),
November 12-16, 2012, Shanghai,
China. Best Paper Award.
Placement techniques for human
units
Usually at design time the developer/designer decides
Where to put human units
Where some triggers should be put in order to invoke
human units if needed
At runtime
Find suitable human units
Invoke human units
Placement of human units
Application-specific
Needs automatic algorithms and supporting tools
ASE Summer 2014 36
Configuring iSCU
Establish „connectedness“ based on compliance
constraints and network topology
Addional cost might occur!
Program SBS and HBS for the iSCU to have a
complete working environment.
Different connectedness
E.g., ring-based, star-based, and master-slave
topologies
Hybrid compute units
38
Hybrid compute unit (HCU): a set of service units
includes software-based services, human-based
services and things-based services that can be
provisioned, deployed and utilized as a collective
on-demand based on different quality, pricing and
incentive models.
Hybrid compute unit (HCU): a set of service units
includes software-based services, human-based
services and things-based services that can be
provisioned, deployed and utilized as a collective
on-demand based on different quality, pricing and
incentive models.
ASE Summer 2014
Hybrid compute unit design –
fundamental elements
39ASE Summer 2014
Hong-Linh Truong, Hoa Khanh Dam, Aditya Ghose, Schahram Dustdar "Augmenting Complex Problem Solving with
Hybrid Compute Units",9th International Workshop on Engineering Service-Oriented Application (WESOA's 2013), In
conjunction with ICSOC 2013, Dec 2, 2013, Berlin, Germany, (c)Springer-Verlag
Hong-Linh Truong, Hoa Khanh Dam, Aditya Ghose, Schahram Dustdar "Augmenting Complex Problem Solving with
Hybrid Compute Units",9th International Workshop on Engineering Service-Oriented Application (WESOA's 2013), In
conjunction with ICSOC 2013, Dec 2, 2013, Berlin, Germany, (c)Springer-Verlag
Hybrid compute unit design --
Relationships
40
Relationship Type HBS SBS TBS HCU
Similarity Yes Yes Yes Yes
Composition Yes Yes Yes Yes
Data Dependency Yes Yes Yes Yes
Control Dependency Yes Yes Yes Yes
Location Dependency Yes Yes Yes Yes
Forwarding Yes Yes No Yes
Delegation Yes Yes No Yes
Social Relation Yes No No Yes
Elasticity Yes Yes No Yes
ASE Summer 2014
Towards programming framework
for HCU
41
Coordination and
Composition Models
Coordination and
Composition Models
[ICSOC12]
ASE Summer 2014
Illustrating Example
Evaluating quality of data (QoD) in
simulations/data analytics processes
Typical components in simulation workflows/data
analytics
Goal using hCAS to support quality of data
control of the simulations/data analytics
Reasons: reducing wasting time and computational
cost, dealing with tacit quality evaluation, etc.
42
pre-processing
solvingpost-
processingdata
analysis
ASE Summer 2014
HCU and simulations
43
Only SBS is needed,
for example, in the QoD evaluation step
before pre-processing
SBS or HBS is used interchangeably or in
combination
for example, in the QoD evaluation after pre-
processing,
Only HBS is used
e.g., in QoD evaluation after solving.
ASE Summer 2014
Turkalytics
Develop „Interaction
Model“ for human
activities
Monitor and analyze
metrics, e.g.,
performance and
location
ASE Summer 2014 46
Paul Heymann and Hector Garcia-Molina. 2011. Turkalytics: analytics for human computation. In Proceedings of the 20th international
conference on World wide web (WWW '11). ACM, New York, NY, USA, 477-486. DOI=10.1145/1963405.1963473
http://doi.acm.org/10.1145/1963405.1963473
Paul Heymann and Hector Garcia-Molina. 2011. Turkalytics: analytics for human computation. In Proceedings of the 20th international
conference on World wide web (WWW '11). ACM, New York, NY, USA, 477-486. DOI=10.1145/1963405.1963473
http://doi.acm.org/10.1145/1963405.1963473
Turkalytics
Track client
detail at client
side (in browser)
Store tracked
information in a
log server
Analyze events
ASE Summer 2014 47
Paul Heymann and Hector Garcia-Molina. 2011. Turkalytics: analytics for human computation. In Proceedings of the 20th international
conference on World wide web (WWW '11). ACM, New York, NY, USA, 477-486. DOI=10.1145/1963405.1963473
http://doi.acm.org/10.1145/1963405.1963473
Paul Heymann and Hector Garcia-Molina. 2011. Turkalytics: analytics for human computation. In Proceedings of the 20th international
conference on World wide web (WWW '11). ACM, New York, NY, USA, 477-486. DOI=10.1145/1963405.1963473
http://doi.acm.org/10.1145/1963405.1963473
VieCOM
SCU execution model and lifecycle
management
Metrics for ICUs and SCUs
Mirela Riveni, Hong-Linh Truong, and Schahram Dustdar, On the Elasticity of Social Compute
Units, CAISE 2014
Mirela Riveni, Hong-Linh Truong, and Schahram Dustdar, On the Elasticity of Social Compute
Units, CAISE 2014
Exercises
Read mentioned papers
Analyze pros and cons of existing frameworks
for data analytics
Survey existing algorithms for matching human
units to data analytics tasks
Examine requirements for locating places for
human units and implement some algorithms
Examine monitoring techniques for cloud of
human compute units
ASE Summer 2014 49
50
Thanks for your attention
Hong-Linh Truong
Distributed Systems Group
Vienna University of Technology
http://dsg.tuwien.ac.at/staff/truong
ASE Summer 2014