Methodologies evaluation

Methodologies evaluation

Agentlink IIIAOSE TFG

Budapest, 17 sep. 2005

AOSE TFG Budapest meeting, 17/9/2005 2

Evaluation framework for AOSEM

Towards an evaluation framework for AOSEM Previous approaches Questionnaire results Review Outline and plan for document on AOSEM evaluation

framework


An evaluation framework for AOSEM

Context Diverse scope of application of methodologies

• Several aspects: analysis, design, implementation, deployment, validation, verification, etc.

• Several application domains: from closed systems to open systems, web support, etc.

Tool support• Tools for modelling and code generation• Some methodologies have no tool support at all (or in a very

experimental state) Development process not always defined Different notations Different agent concepts Standardization efforts Several approaches for integration:

• A common standard agent specification language: which one?• Fragments: method engineering


An evaluation framework for AOSEM

Evaluation of AOSEM can help towards the success of AOSE Clarification of concepts => towards some standardization Integration of fragments Definition of AOSE processes: heavy to light approaches Promotion of tools


Inputs for AOSEM evaluation

A. Sturm, O. Shehory, D. Dori (2004). Evaluation of Agent-Oriented Methodologies. In: AL3 TF1-AOSE TFG

Q.N. Tran, G. Low (2005). Comparison of ten agent-oriented methodologies. In: Henderson-Sellers, B. and Giorgini, P., editors (2005). Agent-Oriented Methodologies. Idea Group Publishing. Chapter XII, pp. 341-367.

C. Bernon, et al. (2004). A Study of some Multi-Agent Meta-Models. Proc. AOSE 2004 (to appear in LNCS, Springer-Verlag).

L. Cernuzzi, G. Rossi (2004). On the evaluation of agent oriented methodologies. In: Proc. of the OOPSLA 2002 Workshop on Agent-Oriented Methodologies.

L. Cernuzzi, M. Cossentino, F. Zambonelli (2005). Process Models for Agent-Based Development. International Journal on Engineering Applications of Artificial Intelligence (EAAI). Elsevier. (in edition?)


Questionnaire

Originally from Mickael Winikoff and modified by Massimo Cossentino

Aim: assess an AOSE methodology against a range of criteria. The criteria fall into a number of areas. Concepts/properties: The ideas that the methodology deals

with, basically the ontology Modelling: The models that are constructed and the

notations used to express the models. Process: The phases and steps that are followed as part of

the methodology. Pragmatics: Practical issues that are concerns when

adopting a methodology (e.g., the availability of training materials and courses, the existence and cost of tools, etc.)


Questionnaire

Answers from: ADELFE (Carole Bernon/creator) INGENIAS (Jorge Gómez-Sanz & Juan Pavón/creators) OPEN Process Framework (OPF) (Brian

Henderson-Sellers/creator) Prometheus-ROADMAP (Lin Padgham/creator) Gaia (Giancarlo Fortino/Alfredo Garro: users!!!) PASSI (M. Cossentino:creator, L. Sabatucci, V. Seidita/PhD

Students: users/doing research on it, 8 graduating students: users)

TROPOS (3 students)

• Others are always welcome!!!• Answers from users (not creators) can provide a better

critical view of methodologies


Questionnaire

Looking at the results of the questionnaire

It can be useful to consider changes in the questionnaire Subjective interpretation of questions and answers Not applicable Missing questions Useful? Clarifying? Identification of methodology challenges

Let’s see what are the results and discuss…

AOSE TFG Budapest meeting, 17/9/2005 9N: None L: Low M: Medium H: High

Questionnaire – Concepts & PropertiesConcept/Property Adelfe Gaia Ingenias OPF PASSI Prome-

theusTROPOS

Autonomy H H H H H/H/M H L

Mental attitudes L N H H L/L/M M M

Proactiveness M L H H H/M/H H N

Reactiveness H L H H H/H/H H N

Concurrency H M H L H/H/M H L

Teamwork and roles

L H H H M/H/H L M

Cooperation model AMAS th. Teamwork ALL ALL Task del./ Teamwork

none Negotiation/ Task del.

Protocols support H H H H H/M/H H N

Communication modes

ALL Async mess. ALL ALL Direct N

Communication language

ALL ACL like ALL ALL Speech acts

messages

Situatedness H H H H H/M/M H H

Environment type All episodic DynamicContinuous

All discrete ALL ALL ALL Inacc., Non episodic, Dynam.

Creator/PhD Students/Grad. Stud.

AOSE TFG Budapest meeting, 17/9/2005 10SD: Strongly Disagree D: Disagree N: Neutral A: Agree SA: Strongly Agree

Questionnaire – Concepts & Properties

Concept/Property Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS

Other agent features

— Opennes Opennes Opennes Mobility, openness, security

Plans, agent decisions

Security, Trust, Delegation, Ownership,

Dependency, Provision

Non supported features

— — Security & Mobility (not

explicitly)

Security & Mobility (on going work)

Complex design-time

social organiza-

tions

Security & Mobility

Dynamic Behavior of

Agent

Clear concepts A A SA SA SA/N/N A A

Overloaded concepts

N D D SD D/D/N D N

More Agent-oriented than OO

A SA SA both SA/A/A SA SA

(Main) Supported agents

Cooperative BDI (mainly) BDI (mainly)

Mainly: State-based,

rational, reactive

ALL BDI, Rational

Society of agents modelling

No SA SA (on going work)

A/-/- No A

Society structure - - Groups/WF - p2p, simple hierarchies,

holons

- Agent Society Pattern, such

as Broker, Mediated,

Matchmaker


Questionnaire – Modelling & Notation

Notation Adelfe Gaia Ingenias OPF PASSI Prome-theus

TROPOS

Support for static (structure) and dynamic (processing) aspects

SA A SA — SA/A/A SA D

Symbols and syntax well defined

A N SA — A/A/D A N

Well defined semantics A D SA — A/N/D A AClear notation A A N — A/A/N A NEasy to use notation A A SA — A/A/N A SAEasy to learn notation N SA A — N/N/N NA N

SD: Strongly Disagree D: Disagree N: Neutral A: Agree SA: Strongly AgreeNA: Not Applicable

A methodology is really notation independent. Yes, there is a need for a modelling language and in the FAME project we have FAML (FAME modelling language) although not yet a notation. So we can’t really answer these notation specific questions (i.e. 21-26)

Creator/PhD Students/Grad. Stud.


Questionnaire – Modelling & Notation

Modelling Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS

Multiple views A N SA SA SA/A/A — A

Adequate and expressive

A N SA SA A/A/N — N

Traceability between models and between models and code

A D SA SA SA/N/A — D

Guidelines and techniques for consistency checking

A SD N N N/N/N — D

Supports refinement SA N SA N SA/A/A — A

Supports modularity SA D A SA SA/A/A — N

Supports component reusability

SA SD SA SA SA/A/A — SD

Extensible SA SD SA SA A/-/- SA A

Supports hierarchical modelling and abstraction

SA D SA SA SA/N/A SA A

Other issues

SD: Strongly Disagree D: Disagree N: Neutral A: Agree SA: Strongly Agree


Questionnaire – Process

Lifecycle coverage Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS

Planning CE CEH CEHRequirements analysis CE CE CEH CEH CEH/-/- CEH CEArchitectural (or agent society) design

CE CE CEH CEH CEH/-/- CEH CE

Detailed (agent) design CE CE CEH CEH CEH/-/- CEH CEImplementation CEH CEH CEH/-/- E PTesting/Debugging H P CEH H PCEHDeployment P CEH CE

Maintenance CEH PDeath CEH

C: Clear definition of activitiesE: Examples givenH: Heuristics givenP: Partial


Questionnaire – Process

Process Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS

Addresses Quality Assurance D SD N SA N/N/A A SDEstimating guidelines (cost, …)

— SD A N D/N/A N N

Support for decision making (e.g. when to move between phases)

A SD SA A N/A/A N D

Development approach Iterative/

incremental

Top-down

Iterative/incremen

talTransformation & architect

ural based

ANY Iterative/Incre-

mental

Iterative/Incre-

mental/Spiral

Top Down

Supports patterns or reusability

A SD D SA SA/-/- N N

Degree of user implication (i.e. it does requires user-designer communication ?)

Medium

— Medium Strong Weak — M



Questionnaire – Pragmatics


Software tools Adelfe Gaia Ingenias OPF PASSI Prome- theus

TROPOS

Diagram editor OpenTool IDK editor PTK GR-Tool, ST-Tool, TAOM4E

Code generator IDK code g. Agent Factory

Design consistency checker

IDK ATA Prototype PTK GR-Tool, ST-Tool

Project Management AdelfeToolkit

Rapid prototyping

Reverse engineering Agent Factory

Automatic testing

Commercial or research product

OT: comm.AT: free

Research Research Research

Research Research

Adequate level of functionalities

A A A/-/- A? N

Quick and easy to learn N A N/-/- A? A

Support in raising the quality

A A SA/-/- SA? N

Reduces time to design/implem.

A SA SA/-/- SA? A

Other comments GPL license, UML/ Ingenias

notation

Considering other tools


Questionnaire – Pragmatics

Pragmatics Adelfe Gaia Ingenias OPF PASSI Prome-theus

TROPOS

Audience All All All All All/-/- All Grad. st., experts,

researcher

Complexity compared to UML/RUP

About the same

About the same

About the same

A lot simpler About the same/-/-

About the

same

Simpler

Resources: Papers

X X X X X X

Text books X X X

Tutorial notes X X X X

Consulting services X X

Training services X X

Nr.applications built with meth.

1-5 21+ 6-20 21+ in OO/ME 21+/-/- 21+ 1-5

Were applications real? Yes No No All Y/Y/N Yes Yes

Any developed by other users?

No Yes Yes Yes Y/Y/N Yes No

Target any specific domain Complex systems

No No All but RT No/-/- No No

Support scalability Yes No Yes Yes A/N/- — N

Supports distributed systems

Yes — Yes Yes SA/A/SA — N


Evaluation framework revisited

Taking the experience of this questionnaire Review evaluation framework criteria and their organization Review method for evaluation: questionnaire, case studies

development, ...• Refine questionnaire• Define case studies

Review metrics• How to avoid subjectivity


Pragmatics

Evaluation framework revisited Criteria for AOSEM evaluation

Features

Domain

Complexity

Process

DeliverablesActivities

Team work

Domain specific methods

Tools

Modelling

Autonomy, society, …

AbstractionModularity

Domain specific concepts

Knowledge skillsScalability


Towards an AOSEM evaluation framework

The evaluation framework should allow: Criteria refinement and extensions Criteria metrics depending on the domain

• E.g. agents in a web service or in robotics Definition of standard case studies for evaluation

• Evaluation of documentation and filling questionnaires is not enough

…


Towards an AOSEM evaluation framework

The framework can be based on the definition and use of evaluation models Case studies for putting the methodologies to work Organized by criteria

• For each criteria, define metrics• Criteria can be refined to get more insight or being more specific

• For instance, agent behaviour, depending on whether BDI, neural network, CBR, reactive, or whatever model is used

• New criteria can be added• Some criteria may be considered non applicable• Associate criteria to case studies


Outline and plan for document on AOSEM evaluation framework

Outline Participants Plan

Documents

Methodologies evaluation