Ghent, July 6, 2004 1

Evaluating a COSMIC-FFP Measurement Procedure for Multi-Layer Object-Oriented

Conceptual SchemasSimon Claeys(Master student Ghent University)Geert Poels(Ghent University & Katholieke Universiteit Leuven)

2Ghent, July 6, 2004

Outline

Research contextDevelopment of a COSMIC-FFP measurement

procedure for MERODE/OOWS Problem statement

How to evaluate this procedure? Possible solution

A laboratory experiment to test the procedure according to Abrahão’s general evaluation model for FSM methods

3Ghent, July 6, 2004

Research context – COSMIC-FFP

Any artifact specifying FUR in whatever form

Mapping

Measuring

FUR in the form of the COSMIC-FFP meta-model

COSMIC-FFP 2.2 FSM method

x CFSU (COSMIC-FFP v2.2)

COSMIC-FFP meta-model

Mapping principles and

rules

Measuring principles and

rules

“A more systematic and detailed procedure would provide precise mapping rules for a larger collection of highly specific artifacts, thus diminishing the level of ambiguity when generating the COSMIC-FFP generic software model. Such a procedure would, by definition, be highly dependent on the nature of the artifacts, which, in turn, depends on the software engineering methodology in use in each organization.” (COSMIC-FFP 2.2 Measurement Manual, p. 30)

4Ghent, July 6, 2004

Research context – Layered OO Conceptual Model

BUSINESS DOMAIN MODEL

FUNCTIONALITY MODEL

PRESENTATION MODEL NAVIGATION MODEL

MERODE

MERODE

OOWS

5Ghent, July 6, 2004

Research context – Measurement procedure

Mapping of concepts in the MERODE/OOWS meta-models onto the COSMIC-FFP meta-model

COSMIC-FFP 2.2 mapping rules4 rules for partial business domain model4 rules for business domain model7 rules for functionality model5 rules for navigation model

6Ghent, July 6, 2004

Problem statement:How to evaluate this procedure ?March and Smith’s research framework for IT

Research activities

Design science Natural science

Build Evaluate Theorize Justify

Research outputs

Constructs

Model

Method

Instantiation

"a set of steps (an algorithm or guideline) used to perform a task. Methods are based on a set of underlying constructs (language) and a representation (model) of the solution space.” (March and Smith 1995, p. 257)

7Ghent, July 6, 2004

Possible solution – Abrahão’s model

The procedure (rules) should enable the task (applying COSMIC-FFP) to be performed with less effort and/or improve the quality of the result compared to applying COSMIC-FFP without this procedure

TaskInputs Outputs

Method

reduce inputs (increase efficiency)

improve outputs (increase effectiveness)

8Ghent, July 6, 2004

Performance => Perceptions => Intentions => Behaviour

Perceived Ease of Use

Perceived Usefulness

Intention to Use

Actual Efficiency

Actual Effective-

ness

PERFORMANCE PERCEPTIONS

BEHAVIOUR

Actual Usage

INTENTIONS

Actual Efficacy Perceived Efficacy Adoption in Practice

9Ghent, July 6, 2004

Operationalize performance properties using ISO/IEC TR 14143-3:2003 Repeatability and reproducability Accuracy Convertibility Discrimination threshold Applicability to functional domains

A laboratory experiment is being planned Spring 2005 Engineers enrolled in a post-graduate Master program in Industrial Management, specialising in ICT (at K.U.Leuven) FSM with COSMIC-FFP as part of ICT Project Management course Only a minority has studied MERODE

10Ghent, July 6, 2004

Possible solution – experimental design Test principle

FURConceptual

schema

results results

COSMIC-FFP Proposed measurementprocedure

MERODE/OOWS

COMPARE

11Ghent, July 6, 2004

Choices to be made: Between-subjects versus within-subjects Choice of application

Reference model (ISO/IEC TR 14143-4:2002) Semantically correct MERODE case-study COSMIC-FFP case-studies

Form of the FUR (in the control group) Text ERD, DFD, UI prototype MERODE/OOWS schema

Data analysis and interpretation How to evaluate accuracy ? (calibration if systematic errors) How to measure reliability ? How relevant is measurement time ? How to interpret perception-based variables ?

12Ghent, July 6, 2004