Upload
ranulfo
View
45
Download
0
Tags:
Embed Size (px)
Citation preview
MACBETH
CARLOS A. BANA E COSTA
Centre for Management Studies of Instituto Superior T�ecnicoTechnical University of Lisbon, Av. Rovisco Pais
1049-001 Lisbon, Portugal
JEAN-MARIE DE CORTE* and JEAN-CLAUDE VANSNICK†
Centre de Recherche Warocqu�e
Universit�e de Mons, Place du Parc, 207000 Mons, Belgium
*[email protected]†[email protected]
This paper presents an up-to-date comprehensive overview of the MACBETH approach to
multicriteria decision-aid. It requires only qualitative judgements about di®erences of attrac-tiveness to help a decision maker, or a decision-advisory group, quantify the relative value of
options. The approach, based on the additive value model, aims to support interactive learning
about evaluation problems and the elaboration of recommendations to prioritize and select
options in individual or group decision making processes. A case study based on a real-worldapplication of MACBETH for multicriteria value measurement of IT solutions is presented. It
shows how the M-MACBETH decision support system can be used in practice to construct an
additive evaluation model. The paper addresses key issues related to structuring the model,
building value scales, weighting criteria and sensitivity and robustness analyzes. Reference isalso made to applications of MACBETH reported in the scienti¯c literature.
Keywords: MACBETH; qualitative value judgements; multicriteria analysis.
1. Introduction
MACBETH (Measuring Attractiveness by a Category-Based Evaluation Tech-
nique)13 is a decision-aid approach to multicriteria value measurement.63,64,90 The
goal behind its conceptualization is to allow measurement of the attractiveness or
value of options through a non-numerical pairwise comparison questioning mode,
which is based on seven qualitative categories of di®erence in attractiveness: is there
no di®erence (indi®erence), or is the di®erence very weak, weak, moderate, strong,
very strong, or extreme? The key distinction from numerical value-measurement
procedures, such as the simple multi-attribute rating technique, or SMART
approach,47,48 is that MACBETH uses only such qualitative judgements of di®erence
in attractiveness in order to generate, by mathematical programming, value scores
International Journal of Information Technology & Decision Making
Vol. 11, No. 2 (2012) 359�387°c World Scienti¯c Publishing CompanyDOI: 10.1142/S0219622012400068
359
for options and weights for criteria. Since the early 1990s, the initial mathematical
formulation was revised and a ¯rst software application implementing MACBETH
was developed.27 This software was later replaced by the current M-MACBETH
decision support system (see Ref. 2 and www.m-macbeth.com) that respects the
theoretical foundations already established,12 allows modeling challenges arising
from practice to be addressed (for instance, it allows for hesitation in choosing
between two or more consecutive categories, except indi®erence) and supports the
\on-the-spot" creation of a computer-based additive value model and sensitivity and
robustness analyzes of the model outputs.
The current paper presents an up-to-date comprehensive overview of MACBETH
that is consistent with the original ideas presented by Bana e Costa and Vansnick
(see Ref. 28). As they observed, decision making in private and public organizations
is above all a human activity in which value judgements of managers and other
actors about the desirability or attractiveness of organizational decision opportu-
nities and alternative courses of action play a crucial role. An important research
challenge is therefore the integration of information technology and human decisions
by means of the design and use of decision-support techniques and systems to answer
the key question of how to elicit and numerically represent value judgements. It is
also an opportunity to avoid the risk of decision science becoming simply a closed
branch of pure mathematics. The existence of a theoretical base is a desirable but not
su±cient condition for the legitimization of a decision-aid theory; it must also be
subjected to practical validation. Moreover, visual appeal and user-friendly \black-
box" software,71 often based on theoretically weak technical procedures, are not the
right answer to the above question: on the contrary, they are a trap for managers and
decision makers in general. Decision-aiding tools need to be simultaneously seman-
tically meaningful, practically operational (user-friendly) and theoretically well
founded.
The MACBETH decision-support approach and software make a contribution in
this direction. Indeed, the M-MACBETH decision support system was designed to be
used by a consultant (facilitator or decision analyst) at di®erent multicriteria
modeling stages (Fig. 1), following the constructivist principles of process consul-
tation.88 This is a socio-technical process that combines the technical elements of
MACBETH with the social aspects of decision conferencing.79,80
Section 2 is devoted to the presentation of the MACBETH questioning and
technical procedure for value elicitation. Section 3 addresses a case study based
on a real-world application of MACBETH for multicriteria value measurement of
IT solutions. It shows how M-MACBETH can be used to construct an additive
evaluation model based on qualitative value judgements of di®erence in attractive-
ness. Key issues of interactive value modeling are discussed. First, how to de¯ne
and operationalize evaluation criteria (the structuring phase, see Sec. 3.2). Second,
how to construct interval value scales which enable one to score the options
on each criterion (the intra-criteria modeling component of the evaluation
phase, see Sec. 3.4). Third, and before or after scoring, how to weight the criteria
360 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
(the inter-criteria modeling component). Weights can be derived by applying the
MACBETH procedure to a set of reference, hypothetical, options (see Sec. 3.3).
Scoring the options on an interval scale within each criterion is important because it
permits one to meaningfully take a weighted average of each option's scores on the
criteria. These overall scores provide answers to the question of how to measure the
relative attractiveness of the options across all the criteria (the synthesis modeling
component). Finally, how to validate the model in the face of uncertainty by using
the software to perform, throughout the modeling process, sensitivity and robustness
analyzes of the model outputs (see Sec. 3.5). As observed by Phillips and Bana e
Costa (see Ref. 80, p. 55), \extensive sensitivity analyzes show that many dis-
agreements or uncertainties in the data make no di®erence to the overall results, and
gradually a sense of common purpose emerges". The paper concludes with some
re°ections in Sec. 4 and includes references to selected real-world applications of
MACBETH reported in the scienti¯c literature.
2. The MACBETH Procedure
2.1. Ordinal and cardinal value information
Let X be a ¯nite set of elements ��� di®erent options or performance levels under
evaluation. Ordinal measurement of the attractiveness (or desirability) of the
elements x of X consists in associating each x with a numerical score ��� a real
number vðxÞ ��� that satis¯es the ordinal measurement conditions (1) (the condition
of strict preference) and (2) (the condition of indi®erence):
8 x; y 2 X : ½x is more attractive than y ðxPyÞ , vðxÞ > vðyÞ�; ð1Þ8 x; y 2 X : ½x is as attractive as y ðxIyÞ , vðxÞ ¼ vðyÞ�: ð2Þ
Such a numerical scale v : X ! < : x ! vðxÞ can be constructed by asking for
ordinal value information from an individual or group evaluator (DM). That is, the
DM is asked for a ranking of the elements of X in order of decreasing attractiveness
Fig. 1. Phases of the MACBETH decision-aiding process.
MACBETH 361
(with the possibility of indi®erence). If this is done for each one of several criteria,
Arrow's impossibility theorem1 shows that, unfortunately, aggregating individual
rankings into an overall ranking always implies some form of arbitrariness.
Cardinal measurement of the attractiveness of the elements x of X consists in
associating each x with a numerical score ��� a real number vðxÞ ��� that satis¯es not
only the ordinal conditions (1) and (2) but also the additional condition (3):
8w; x; y; z 2 X with x more attractive than y and w more attractive than z :
the ratio ½vðxÞ � vðyÞ�=½vðwÞ � vðzÞ� measures the difference in attractiveness
between x and y when the difference in attractiveness between w and z is
taken as the measurement unit: ð3ÞSuch a numerical scale v : X ! < : x ! vðxÞ can be constructed by positioning
the elements of X on a vertical axis so that:
(1) 8 x; y 2 X : x is positioned above y if and only if x is more attractive than y
(ordinal value information)
(2) the relative distances between the elements de X on the vertical axis re°ect the
relative di®erences in attractiveness between these elements (cardinal value
information).
A scale v that satis¯es the measurement conditions (1)�(3) is an interval scale of
measurement, that is, v is a numerical scale unique up to a positive linear (or a±ne)
transformation.
Several elicitation procedures can be conceived in order to obtain cardinal value
information. One could ask the DM to provide, 8w; x; y; z 2 X with x more attractive
than y and w more attractive than z, a direct numerical estimation of the ratio of the
di®erences in attractiveness between x and y on the one hand and between w and z
on the other hand. This mode of questioning is far from straightforward,90 however.
Moreover, the number of questions dramatically increases with the number of
elements of X . One could reduce the number of questions by taking the di®erence
between two arbitrarily chosen elements of X as a reference unit of measurement and
ask the DM to estimate, 8 x; y 2 X , the number of times Eðx; yÞ the di®erence
between x and y is greater or smaller (if not equal) to the reference di®erence. It
would, however, be surprising if the judgements made were so perfectly consistent
that they determined an interval value scale. In the procedure proposed by Kirkwood
(see Ref. 64) the smallest of the di®erences between consecutive elements in a ranking
is identi¯ed by the DM that subsequently numerically rates the number of times each
one of the remaining consecutive di®erences is greater (if not equal) to the smallest
one. Alternatively, the direct rating technique, used in SMART, requires three main
tasks to be performed: (1) selection of two reference elements for the rating scale;
(2) assignment of numerical values to these reference elements, usually 100 and zero,
respectively; and (3) asking the DM to assign to each one of the remaining elements a
numerical value that denotes its relative attractiveness with respect to the two
362 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
references. The consistency of the rating scale is tested in such a way that di®erences
between numerical ratings should re°ect (i.e., measure) di®erences of attractiveness
for the DM.
With the deliberate intent of o®ering a way to avoid the eventual di±culty90 and
cognitive uneasiness50 experienced by evaluators when trying to express their pre-
ference judgements numerically, in MACBETH the transition from ordinal to car-
dinal information is facilitated by a non-numerical pairwise comparison questioning
mode, according to which qualitative judgements, instead of quantitative ones, are
elicited from the DM and are the basis on which a interval value scale will be
constructed, progressively and interactively with the DM, as explainsed in detail in
Sec. 2.2.
2.2. The MACBETH procedure for obtaining cardinal information
2.2.1. Obtaining ordinal information
One can start by asking the DM to rank the elements of X by decreasing attrac-
tiveness. Alternatively, when this is found di±cult, one can ask the DM to compare
the elements two at a time: is one of the two elements more attractive than the other
and if yes, which one? For each paired comparison, M-MACBETH tests the com-
patibility of the ordinal information provided by the DM with the existence of a
ranking and, if an incompatibility is detected, it displays the source of inconsistence
graphically and makes suggestions to overcome it. Figure 2 shows an example of an
inconsistent set of ordinal judgments. A \P" in an entry in the matrix of judgments
means that the DM judges the element in the row to be more attractive than (i.e.,
strictly preferable to) the element in the column, whereas an \I" in an entry means
that the two elements are judged to be equally attractive (indi®erent). The three
judgements bId, dIe and bPe, represented in the graph on the right in Fig. 2, are not
consistent and three alternative suggestions are presented in the respective cells in
the matrix: switching one of the two indi®erences to strict preference (indicated by
an arrow up icon) or switching the strict preference to indi®erence (indicated by an
arrow down icon).
Fig. 2. Example of ordinal inconsistency and suggestions for obtaining a ranking.
MACBETH 363
2.2.2. Obtaining pre-cardinal information
When x is more attractive than y ðxPyÞ, the DM is asked for a qualitative judgement
about the di®erence of attractiveness between x and y, by presenting the DM with six
categories: very weak di®erence (category C1Þ, weak (C2Þ, moderate (C3Þ, strong(C4Þ, very strong (C5Þ and extreme (C6Þ. Judgemental disagreement or hesitation
between two or more consecutive categories, except indi®erence, is also allowed. The
questions may be asked in any sequence and can be stopped at any moment: if xPy
and no MACBETH judgement of di®erence in attractiveness was elicited for the
ordered pair ðx; yÞ, M-MACBETH assigns it to the union of all six categories
(a \positive" di®erence of attractiveness, meaning that, for that pair, the information
available is merely ordinal). That is, for a set X of n elements, it is not necessary to
perform all of the nðn � 1Þ=2 paired comparisons and populate the upper triangular
part of the MACBETH matrix completely (see Fig. 3). Indeed, the minimal number
of judgements required is n � 1, e.g., comparing one element with the remaining ones
or comparing the elements rank-ordered consecutively (the same number of paired
comparisons as in the above-mentioned Kirkwood's numerical procedure). As
emphasized in Ref. 17, however, it is good practice to ask for additional judgements
to perform \a number of consistency checks" (Ref. 90, p. 228), e.g., by ¯lling in the
two ¯rst diagonals of the matrix, as suggested by Belton and Stewart (Ref. 31, p.173)
and exempli¯ed in Fig. 3. This implies a total of 2n � 3 judgements, whereas ¯lling
the borders of the upper triangular matrix, as done in Ref. 8, requires 3n � 6 paired
comparisons. Each time that a qualitative judgement is elicited, M-MACBETH tests
the consistency of all the judgements made by the DM, that is, their compatibility
with cardinal information, verifying whether it is possible to associate with each
element x of X a number vðxÞ satisfying conditions (4)�(6):8 x; y 2 X : xIy ) vðxÞ ¼ vðyÞ; ð4Þ8 x; y 2 X : xPy ) vðxÞ > vðyÞ; ð5Þ8 x; y 2 Ci [ � � � [ Cs
and 8w; z 2 Ci 0 [ � � � [Cs 0 with i; s; i 0; s 0 2 f1; 2; 3; 4; 5; 6g;i � s and i 0 � s 0 : i > s 0 ) vðxÞ � vðyÞ > vðwÞ � vðzÞ: ð6Þ
That is, the judgements of the DM are such that the elements of X can be
positioned in a vertical axis in such a way that:
. 8 x; y 2 X : (the DM judged x and y equally attractive)) (x and y are coincident)
. 8 x; y 2 X : (the DM judged x more attractive than y)) (x is positioned above y).
. 8 x; y;w; z 2 X with x more attractive than y and w more attractive than z, if it
results from the judgements of di®erence in attractiveness given by the DM that
the di®erence of attractiveness between x and y is greater than the di®erence of
attractiveness between w and z, then the distance between x and y is greater than
the distance between w and z.
When such a scale exists, the information assessed is called pre-cardinal value
information. Conversely, if inconsistency is detected, i.e., if conditions (4)�(6)
364 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
cannot be simultaneously veri¯ed by the set of judgements elicited, suggestions are
o®ered to resolve it and render the information pre-cardinal. Figure 3 shows an
example with ¯ve elements, a; b; c; d, and e. Suppose that the DM started by ranking
the elements as follows: aPbPcPdIe. Then the DM pairwise compared the elements
consecutive in this ranking by assigning each pair to a category of di®erence in
attractiveness, i.e., ða; bÞ 2 C1 (very weak di®erence), ðb; cÞ 2 C2 [ C3 (hesitation
between weak and moderate di®erence ��� the \weak-mod" entry in the matrix of
Fig. 3), ðc; dÞ 2 C2 (weak) (as d and e were judged as indi®erent, the software
indicates there is \no" di®erence of attractiveness between them). Next, element \a"
was judged strongly more attractive than \c". All these judgements are consistent,
but the software pointed out an inconsistence once the moderate judgement between
\b" and \d" was entered in the matrix. Indeed, these judgements imply, given
condition (6), that vðaÞ � vðbÞ > vðcÞ � vðdÞ and vðbÞ � vðdÞ > vðaÞ � vðcÞ, whichare incompatible inequalities because, by summation, vðaÞ � vðdÞ > vðaÞ � vðdÞ.Four alternative modi¯cations of only one category are presented in the respective
cells in the matrix: to move ða; bÞ or ðb; dÞ one category up (indicated by an arrow up
icon) or to move ða; cÞ or ðc; dÞ one category down (indicated by an arrow down
icon), each one su±cient, if accepted, to make the set of judgements verify conditions
(4)�(6) (all other possible ways that resolve the inconsistency can also be displayed
in M-MACBETH). We suppose hereafter that the DM decided to move ða; cÞ downto the category \moderate", while keeping all the remaining judgements unchanged,
which gives rise to the MACBETH matrix of consistent qualitative judgements shown
in Fig. 4.
Fig. 4. Consistent MACBETH judgements.
Fig. 3. Example of inconsistent MACBETH judgements and suggestions for obtaining pre-cardinalinformation.
MACBETH 365
2.2.3. Pre-cardinal and MACBETH scales
From pre-cardinal information it is always possible to determine a pre-cardinal scale,
that is, a numerical scale that satis¯es conditions (4)�(6). A particular pre-cardinal
scale v, the \basic MACBETH scale", can be derived by M-MACBETH, at any
moment in the interactive modeling process, from the pre-cardinal information eli-
cited. This scale is obtained by solving the following linear programming problem
(LP-MACBETH), where vðxÞ is the score assigned to element x of X , xþ and x� are
two elements of X such that xþ is at least as attractive as any other element of X and
x� is at most as attractive as any other element of X .
LP-MACBETH:
Min½vðxþÞ � vðx�Þ�s:t:
ðc1Þ vðx�Þ ¼ 0 ðarbitrary assignmentÞ:ðc2Þ vðxÞ � vðyÞ ¼ 0; 8 x; y 2 C0:
ðc3Þ vðxÞ � vðyÞ � i; 8 x; y 2 Ci [ � � � [ Cs
with i; s 2 f1; 2; 3; 4; 5; 6g and i � s:
ðc4Þ vðxÞ � vðyÞ � vðwÞ � vðzÞ þ i � s 0; 8 x; y 2 Ci [ � � � [ Cs
and 8w; z 2 Ci 0 [ � � � [ Cs 0 with i; s; i 0; s 0 2 f1; 2; 3; 4; 5; 6g;i � s; i 0 � s 0 and i > s 0:
Conditions c2 to c4 are \conditions of order preservation" (COP): conditions c2 and
c3 ensure the preservation of the order in the DM ranking of the elements and
condition c4 ensures the preservation of the order between di®erences of attrac-
tiveness implicit in the qualitative judgements of di®erent categories expressed by
the DM (note that no condition is imposed for judgements of a same category).
The optimal solution of LP-MACBETH is not necessarily unique and, therefore, to
ensure mathematically the uniqueness of the basic MACBETH scale, supplementary
technical linear programs are used (see Bana e Costa et al.12 for technical details). The
basic MACBETH scale for the set of judgements in Fig. 4 is vðaÞ ¼ 5; vðcÞ ¼ 4;
vðbÞ ¼ 2, and vðdÞ ¼ vðeÞ ¼ 0. Any pre-cardinal scale obtained by a positive a±ne
transformation of the basic MACBETH is an \anchored MACBETH scale". In par-
ticular, the scale anchored on vðxþÞ ¼ 100 ð¼ vðaÞÞ and vðx�Þ ¼ 0 ð¼ vðdÞ ¼ vðeÞÞ isautomatically displayed by M-MACBETH on a vertical axis, as shown in Fig. 5.
2.2.4. From pre-cardinal to cardinal information
The DM is invited to observe the MACBETH scale axis and compare value intervals.
This discussion is an essential ¯nal step in the learning path for obtaining cardinal
information, i.e., of measuring attractiveness in an interval value scale. The quali-
tative information thereto elicited, however, is not su±cient for this purpose and it is
worth noting that many other possible scales would equally well respect the con-
ditions of order preservation c2, c3 and c4. When an element is selected in the axis, an
366 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
interval is displayed, within which the DM can adjust the position of that element
without violating those conditions. For instance, the DM may want to decrease the
numerical value of element c (within the range shown in Fig. 6) so that the scale
interval associated with the qualitative di®erence in attractiveness between b and c
(where the DM hesitated between weak or moderate) becomes greater than the scale
interval associated with the judgement \weak" between c and d. The value of c
cannot be decreased to 20 or less, however, because this would make the interval
between c and d smaller than the interval between a and b that numerically rep-
resents a very weak di®erence of attractiveness. Adjustments in position can be made
for any element, each time keeping the positions of the remaining ones ¯xed, to make
the intervals between the elements re°ect the relative di®erences of attractiveness
between the elements, for the DM (note: should the DM want to position an element
outside its interval, this would only be possible if the position of at least one other
element were changed and conditions c2 to c4 remained inviolate ��� for instance, to
set the value of c below 20 would require a previous reduction of the size of the
interval between a and b by moving up b on the axis). At the end of the discussion of
the MACBETH scale, the ratio between any two positive di®erences of value should
represent the proportion between the respective di®erences in attractiveness. For
example, the ¯nal value of element c ¯xed as 30 in Fig. 6 re°ects that the di®erence
between b and c is half the di®erence between a and d.
Fig. 5. Basic and anchored MACBETH scales.
MACBETH 367
2.3. Determining by hand the basic MACBETH scale
In the case of small consistent sets of judgements, the basic MACBETH scale can be
determined by hand. This is very useful for a facilitator or decision analysis during a
value elicitation session as it allows them to show to the DM and other participants a
clear way to derive a quantitative representation of the DM's qualitative judgements
of di®erence in attractiveness. In this section, X denotes a set of n elements x1; x2;
. . . ; xn previously ranked from left to right by decreasing relative attractiveness, that
is, for r; p 2 f1; 2; . . . ; ng; r < p implies xrðP [ I Þxp, and vðxrÞ denotes the numerical
value of xr . Using these notations, the LP-MACBETH program (see Sec. 2.2.3) can
be rewritten as follows (with Cij ¼ Ci [ . . . [ Cj):
min½vðx1Þ � vðxnÞ�s:t:
ðc1Þ vðxnÞ ¼ 0:
ðc2Þ vðxpÞ � vðxrÞ ¼ 0; 8 ðxp; xrÞ 2 C0 with p < r:
ðc3Þ vðxpÞ � vðxrÞ � i; 8 i; j 2 f1; 2; . . . ; 6g with i � j; 8 ðxp; xrÞ 2 Cij :
ðc4Þ vðxpÞ � vðxrÞ � vðxkÞ � vðxmÞ þ i � j 0; 8 i; j; i 0; j 0 2 f1; 2; . . . ; 6gwith i � j; i 0 � j 0 and i > j 0; 8 ðxp; xrÞ 2 Cij ; 8 ðxk ; xmÞ 2 Ci 0j 0 :
If none of the judgements of the DM are expressed by more than one category (i.e.,
there is no hesitation between two categories for any of the judgements elicited),
Fig. 6. Adjusting the value of an element within the respective range of variation.
368 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
constraints c3 and c4 can be written simply as:
ðc5Þ vðxpÞ � vðxrÞ � vðxkÞ � vðxmÞ þ i � i 0; 8 ðxp; xrÞ 2 Ci
and 8 ðxk ; xmÞ 2 Ci 0 with 0 � i 0 < i � 6:
We will show how to determine by hand the basic MACBETH scale in the case of
no hesitation. Since, from c1, vðxnÞ ¼ 0, one only needs to determine the n � 1
elementary di®erences vðx1Þ � vðx2Þ; vðx2Þ � vðx3Þ; . . . ; vðxn�1Þ � vðxnÞ. The way to
determine these elementary di®erences is to proceed as follows:
. Initialization:
8 ðxp; xrÞ 2 C0; vðxpÞ � vðxrÞ ¼ 0;
S : \empty" system of constraints that will be built progressively.
. Start with � ¼ 1:
˚ 8 ðxp; xpþ1Þ 2 C� with p 2 f1; . . . ; n � 1g : vðxpÞ � vðxpþ1Þ ¼ n� þ k�;pwhere: n� ¼ maxfvðxiÞ � vðxjÞji < j and ðxi; xjÞ 2 C��1g þ 1 and
k�;p ¼ \minimal" positive correction with respect to n� to satisfy S.
¸ 8 i; j 2 f1; . . . ; ng with i < j and j � i � 2, calculate vðxiÞ � vðxjÞ whenever it ispossible.
� Verify c5:
� if c5 is not respected,
add to S the condition(s) that must be satis¯ed by the elementary
di®erences so that c5 is satis¯ed
! � ¼ 1
! ˚
� if c5 is respected,
if � ¼ 6, end!
if � < 6:
if 8 p 2 f1; . . . ; n � 1g; vðxpÞ � vðxpþ1Þ calculated, endif not, set � �þ 1, then go to ˚.
We illustrate this procedure in the case of n ¼ 6 on the basis of the consistent
matrix of judgements in Fig. 7.
Initialization: 8 ðxp; xrÞ 2 C0; vðxpÞ � vðxrÞ ¼ 0; S ¼ �
(1) � ¼ 1:
˚ vðx1Þ � vðx2Þ ¼ vðx4Þ � vðx5Þ ¼ vðx5Þ � vðx6Þ ¼ 1 ðk1;1 ¼ k1;4 ¼ k1;5 ¼ 0
because S is \empty").
¸ vðx4Þ � vðx6Þ ¼ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 2.
� � c5 is respected� � 2.
MACBETH 369
(2) � ¼ 2:
˚ n2 ¼ 3; vðx2Þ � vðx3Þ ¼ 3ðk2;2 ¼ 0 because S is \empty").
¸ vðx1Þ � vðx3Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� ¼ 4.
� � c5 is respected� � 3.
(3) � ¼ 3:
˚ n3 ¼ 5; vðx2Þ � vðx3Þ ¼ 5ðk3;3 ¼ 0 because S is \empty").
¸ vðx1Þ � vðx3Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� ¼ 4;
vðx1Þ � vðx4Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� ¼ 9;
vðx1Þ � vðx5Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ�þ ½vðx4Þ�vðx 5Þ� ¼ 10;
vðx1Þ � vðx6Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx 2Þ � vðx 3Þ� þ ½vðx3Þ � vðx4Þ�þ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 11;
vðx2Þ � vðx4Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� ¼ 8;
vðx2Þ � vðx5Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� ¼ 9,
vðx2Þ � vðx6Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ�þ ½vðx5Þ � vðx 6Þ� ¼ 10;
vðx3Þ � vðx5Þ ¼ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� ¼ 6;
vðx3Þ � vðx6Þ ¼ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 7;
vðx4Þ � vðx6Þ ¼ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 2 (see Fig. 8).
� c5 is not respected: as ðx1; x4Þ 2 C4 and ðx2; x5Þ 2 C3, one must have
vðx1Þ � vðx4Þ � vðx2Þ � vðx5Þ þ 1, that is, in terms of the elementary di®er-
ences:
½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ�� ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� þ 1,
which implies:
vðx1Þ � vðx2Þ � vðx4Þ � vðx5Þ þ 1
S ¼ fvðx1Þ � vðx2Þ � vðx4Þ � vðx5Þ þ 1g thus k1;1 ¼ 1
� ¼ 1:
Fig. 7. Consistent matrix of MACBETH qualitative judgements with no hesitation.
370 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
(4) � ¼ 1:
˚ vðx4Þ � vðx5Þ ¼ vðx5Þ � vðx6Þ ¼ 1 and vðx1Þ � vðx2Þ ¼ 2 ðk1;4 ¼ k1;5 ¼ 0;
k11 ¼ 1Þ:¸ vðx4Þ � vðx6Þ ¼ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � �vðx6Þ� ¼ 2:
� � c5 is respected� � 2:
(5) � ¼ 2:
˚ n2 ¼ 3; vðx2Þ � vðx3Þ ¼ 3 ðk2;2 ¼ 0Þ.¸ vðx1Þ � vðx3Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� ¼ 5.
� � c5 is respected� � 3.
(6) � ¼ 3:
˚ n3 ¼ 6; vðx3Þ � vðx4Þ ¼ 5 ðk3;3 ¼ 0Þ:¸ vðx1Þ � vðx3Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� ¼ 5;
vðx1Þ � vðx4Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� ¼ 11;
vðx1Þ � vðx5Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ�þ ½vðx4Þ � vðx5Þ� ¼ 12;
vðx1Þ � vðx6Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ�þ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 13;
vðx2Þ � vðx4Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� ¼ 9;
vðx2Þ � vðx5Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� ¼ 10;
vðx2Þ � vðx6Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ�þ ½vðx5Þ � vðx6Þ� ¼ 11;
vðx3Þ � vðx5Þ ¼ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� ¼ 7,
vðx3Þ � vðx6Þ ¼ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 8;
vðx4Þ � vðx6Þ ¼ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 2:
� � c5 is respected� 8 p 2 f1; 2; 3; 4g; vðxpÞ � vðxpþ1Þ calculated� end.
Fig. 8. Applying the \hand procedure".
MACBETH 371
As all the elementary di®erences are determined (see Fig. 9), the basic MAC-
BETH scale can be obtained:
vðx6Þ ¼ 0;
vðx5Þ ¼ vðx6Þ þ ½vðx5Þ � vðx6Þ� ¼ 1;
vðx4Þ ¼ vðx5Þ þ ½vðx4Þ � vðx5Þ� ¼ 2;
vðx3Þ ¼ vðx4Þ þ ½vðx3Þ � vðx4Þ� ¼ 8;
vðx2Þ ¼ vðx3Þ þ ½vðx2Þ � vðx3Þ� ¼ 11;
vðx1Þ ¼ vðx2Þ þ ½vðx1Þ � vðx2Þ� ¼ 13:
3. Multicriteria Evaluation with MACBETH: The SOA Case Study
3.1. The context
ANACOM (\Autoridade Nacional de Comunicações"; www.anacom.pt) is the
national regulatory authority for electronic communications and postal services in
Portugal. It also provides advice on public policy to the government and is the
international representative of the Portuguese communications sector. In 2007,
ANACOM adopted the MACBETH approach and software to support option
evaluation and decision making in \beauty contests",37 when awarding telecom
licences, and public procurement processes across the entire corporation, covering
the provision of contracts for the execution or concession of public services and
works, and the acquisition of goods and services. The case study developed in this
section is inspired by the evaluation process of the tenders presented in response to
the international call for tenders for the \acquisition of BPMS-DMS-SOA solution"
(nr. 2/2008) launched by ANACOM and conducted by an internal technical jury
(the DM) in the Information Systems Division. In brief, a BPMS-DMS-SOA solution
includes the delivery of a business process management system (BPMS) and a
document management system (DMS) integrated in a service-oriented architecture
(SOA). It should be a process-centric information technology solution that is able to
integrate people, systems and data41 and make available business functionality or
Fig. 9. Applying the \hand procedure" (continued).
372 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
application logic to system users, or consumers, as shared reusable services.68 Three
IT companies answered the BPMS-GD-SOA call for tenders. In this case study,
performances of their tenders (labeled A, B, and C) and DM judgemental infor-
mation were disguised for con¯dentiality reasons and to simplify or improve the focus
of the presentation.
3.2. Structuring issues
The three criteria de¯ned to evaluate the tenders and announced in the call for
tenders were: the extent to which the solution proposed in each tender was techni-
cally and functionally suitable for achieving ANACOM objectives (criterion Cr1:
\Suitability"), the extent to which each tender presented an adequate execution plan
(criterion Cr2: \Execution") and the total price proposed in each tender including
licensing, technical support and document management services (criterion Cr3:
\Price").
One recommended practice when structuring and operationalizing criteria fol-
lowing the MACBETH approach in a public call for tenders is to make clear to the
potential tenderers what constitutes a good performance (goodjÞ on each criterion
Crj (see Refs. 9 and 26), e.g., good3 ¼ 400 thousand , a price below which a tender
would be assessed as very attractive. The description of references (goals, aspiration
levels, standards) of good performance may increase the chances of receiving good
proposals from potential tenderers. Of course, not all of them, if any, will be able to
achieve good performances in all criteria. Indeed, quality versus cost or risk are
crucial trade-o®s, as it may be necessary to privilege some goal(s) to the detriment of
others. A complementary structuring recommendation of MACBETH is to de¯ne a
reference of neutral performance (neutraljÞ on each criterion Crj , e.g., neutralj ¼ 600
thousand , a price above which a tender will be assessed as unattractive. Once the
two reference performances are entered into the MACBETH model, the scores of 100
and 0 are assigned to them by default and the criterion value scale automatically
displayed by M-MACBETH is anchored on vjðgoodjÞ ¼ 100 and vjðneutraljÞ ¼ 0.
In general, performance on quality and risk criteria depends on a signi¯cant
number of interrelated indicators or characteristics. These are often combined by
means of point systems57 that su®er from the drawback of ignoring that elementary
value dimensions are often not additive-independent.64 As remarked by Edwards and
Barron (Ref. 48, p. 315), \violations of conditional monotonicity usually easy to
detect judgementally mean that additive models should not be used" (to aggregate
interrelated dimensions). Alternatively, multidimensional performance scales (see
Refs. 5, 17 and 60) can be constructed by applying noncompensatory aggregation
procedures such as the \determinants technique" (see Ref. 9):
(1) Establish two references, \satisfactory" and \neutral", in each of the dimensions.
(2) Classify each dimension as \determinant" (D), \important" (I) or \secondary"
(S). A dimension is determinant if a performance that is negative (worse than
neutral) in that dimension is a necessary and su±cient condition for a proposal to
MACBETH 373
be considered negative (worse than neutral) in the set of all dimensions (this
means that a determinant dimension has a noncompensatory nature).
(3) De¯ne a reference of good performance on the set of dimensions as a reference
pro¯le such that all determinant dimensions are satisfactory and a majority of
the important dimensions are satisfactory; and a reference of neutral perform-
ance on the set of dimensions as a reference pro¯le such that a majority of the
determinant and important dimensions are neutral, without any dimension being
negative.
(4) Similar rules can be used to de¯ne other reference performance pro¯les, e.g.,
\better than good" as all determinant and important dimensions are satisfactory
and no secondary characteristic is negative, or \worse than neutral" as at least
one determinant characteristic is negative.
In the SOA case, for example, there were six elementary dimensions related to
ease of implementation of the solution. As indicated in Table 1, only one was con-
sidered determinant, two were viewed as important and the remaining three were
deemed secondary.
3.3. Issues in inter-criteria modeling: MACBETH weighting
It was set in the SOA call for tenders that the overall value score V ðxÞ of each tender
x would be calculated by the simple additive value model (7)
V ðxÞ ¼X3j¼1
kjvjðxÞ; withvjðgoodjÞ ¼ 100
vjðneutraljÞ ¼ 0;
�kj > 1 ðj ¼ 1; 2; 3Þ and
X3j¼1
kj ¼ 1;
ð7Þ
where the parameters kj ðj ¼ 1; 2; 3Þ are scaling constants ��� usually called
\weights"��� enabling one to convert the single-criterion value scores vjðxÞ into unitsof overall value. In this section, we will illustrate how these relative weights can be
determined with the MACBETH procedure. It is worth noting that the numerical
weights assigned to the SOA criteria were set before the tenders were known and
Table 1. Elementary dimensions of ease of implementation.
Elementary Dimensions Type Satisfactory Levels Neutral Levels
Time needed to install the platform in the ANACOM
infrastructure
D 4 h 8 h
Independence of DBA (Database Administrator)
privileges
S Yes No
Disk space required S 5GB 50GB
RAM required S 3GB 6GBSize of the technical team I 2 technicians 3 technicians
Duration of the concept test I 2 days 4 days
374 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
were disclosed in the call for tenders, in compliance with national and European
Union regulations.
Let [Crj �; j ¼ 1; 2; 3, denote three hypothetical tenders, each with performance on
criterion Crj equal to the reference goodj and performance on each one of the two
remaining criteria equal to the respective neutral reference performance; and let
[neutral] denote the hypothetical tender with performance on all criteria equal to the
respective neutral reference performances:
[Cr1] ¼ [good1, neutral2, neutral3],
[Cr2] ¼ [neutral1, good2, neutral3],
[Cr3] ¼ [neutral1, neutral2, good3],
[neutral] ¼ [neutral1, neutral2, neutral3].
It is interesting to observe that for each i ¼ 1; 2 or 3:
V ð½Crj �Þ � V ð½neutral�Þ ¼ kj ½vjðgoodjÞ � vjðneutraljÞ� ¼ 100kj : ð8ÞTherefore, in order to set the weights it is su±cient to elicit cardinal information
concerning the (overall) attractiveness of the four hypothetical tenders [Cr1], [Cr2],
[Cr3], and [neutral]. This can be done by paired comparisons, following the MAC-
BETH procedure described in Sec. 2. Suppose that the DM made the qualitative
judgements shown in Fig. 10, for which M-MACBETH suggested a ¯rst weighting
scale, shown in percentages at the right of the matrix, subsequently adjusted to the
weights shown in percentages in the bar chart: k1 ¼ 0:55; k2 ¼ 0:25 and k3 ¼ 0:20.
This MACBETH weighting procedure is a qualitative alternative to the quanti-
tative \swing weighting procedure" used in SMART, which, as described in Ref. 90
(see also Ref. 48), would consist in asking the DM to rank the hypothetical tenders
(this can be done by paired comparisons, as in Ref. 60, p. 290), assign a \swing
weight" of 100 to the preferred improvement and swing-weight the other improve-
ments as percentages of this.
In practice, a straightforward MACBETH question�answer process to elicit
weighting judgements could develop as follows between the facilitator (F) and
the DM:
F: Suppose there is one tender with neutral performances in all criteria ([neutral]). If
its performance on one criterion could be improved to good, on which criterion would
the improvement from neutral to good be the most important? (note that this
Fig. 10. MACBETH weighting.
MACBETH 375
question is asking on which criterion the di®erence in attractiveness between good
and neutral is the greatest).
DM: The most important improvement from neutral to good would be on suitability.
F: How important would the improvement be from neutral to good suitability?
DM: The improvement from neutral to good suitability has very strong importance.
F: Which would be the next most important improvement from neutral to good, on
execution or on price?
DM: Execution.
F: How important would be the improvement from neutral to good on execution?
DM: Moderately.
F: How much more important would be the improvement from neutral to good on
suitability than the improvement from neutral to good on execution?
DM: The improvement on suitability would be strongly more important than the
improvement on execution.
F: How important is the improvement from neutral to good price?
DM: Moderately.
F: Would the improvements from neutral to good on execution and on price be
equally important?
DM: No, the improvement on delivery would be more important.
F: How much more important?
DM: Very weakly.
F: One last question. Would the improvement from neutral to good on suitability be
strongly or very strongly more important than the same improvement on price?
DM: Strongly.
Note that the facilitator's questions in this dialogue would be misleading if they were
asked simply in terms of the importance of the criteria rather than in terms of the
importance of the improvements from neutral to good on the criteria. That would be
a mistake (unfortunately, \the most common critical mistake" (see Ref. 60, p. 147
and p. 279).
3.4. Issues in intra-criteria evaluation: MACBETH scoring
At the level of each criterion, a MACBETH model can be built following one of two
basic paths: direct evaluation, in which options are compared with one another and a
value score is assigned to each one; or indirect evaluation, in which a value function is
constructed upon a previously de¯ned descriptor of performances and then the value
function is used to transform each option performance in a value score. The pros and
cons of these two paths are discussed by Bana e Costa and colleagues.22
In the SOA case, the tenders were directly compared for Suitability and Execution
(criteria Cr1 and Cr2Þ, whereas for the Price criterion (Cr3Þ a value function was
a priori de¯ned and announced in the procurement documents. The value function is
often assumed to be linear within the Price range, but this does not prevent very high
prices. On the contrary, a set of MACBETH judgements as shown in Fig. 11, with a
376 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
weak di®erence of attractiveness between 400 and 600 thousand (the good and
neutral prices) and an extreme di®erence of attractiveness between 600 and 700
thousand (i.e., prices higher than neutral), giving rise to a two-piecewise linear
value function, reveals the DM's intention to discourage the proposal of high prices
by potential tenderers.
As regards the Execution criterion (Cr2Þ, ¯rst the DM appraised the intrinsic
attractiveness of each tender, by comparing its performance with the good and
neutral reference performances. Then, the tenders and the two references were
pairwise compared, qualitatively, with the MACBETH procedure. Figure 12 shows
an illustrative example of the MACBETH judgements on Execution: an initial
inconsistent matrix, with the software indicating two possible modi¯cations of one
category, and the ¯nal consistent matrix, together with the value scores resulting
from the discussion by the DM. A similar interactive process was followed to score
the tenders on the Suitability criterion (Cr1Þ (see Fig. 13).
The synthesis of all the intra-criteria and inter-criteria cardinal information
so far assessed (see Table 2) was then performed by applying the additive model
V ðxÞ ¼ 0:55v1ðxÞ þ 0:25v2ðxÞ þ 0:20v3ðxÞ to calculate the overall value of each
tender: V ðAÞ ¼ 50:35;V ðBÞ ¼ 53:45 and V ðC Þ ¼ 92:90 (the prices of tenders A, B,
and C were, respectively, 378, 397, and 546 thousand ).
3.5. Issues of sensitivity and robustness analysis
Decision-making processes often involve imprecise data and uncertain information.
Managers are often concerned with the stability of the outputs of the additive value
model. For example, they may be interested in analyzing if the best option would
Fig. 11. Building a value function on Price (Cr3Þ.
Fig. 12. MACBETH intra-criterion evaluation process on Execution (Cr2Þ.
MACBETH 377
change in the face of small variations of the weight of one criterion, e.g., Price. The
sensitivity analysis graph in Fig. 14 shows that C remains the best tender even when
the weight of Price is increased from 20% to nearly 47%. This is highlighted in Fig. 14
by the band of variation of the weight, compatible with the DM judgements indi-
cated in Fig. 10, whose upper bound is around 25%.
Fig. 13. MACBETH judgements and value scale on Suitability (Cr1Þ.
Table 2. Table of scores.
Tenders Value Scores
Suitability Execution Price Overall
C 100 130 27 92.90
B 33 60 101.5 53.45
A 33 40 111 50.35
Weights 0.55 0.25 0.20
Fig. 14. Sensitivity analysis of the weight of Price.
378 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
It may also be interesting to analyze what conclusions can be drawn in the face of
only ordinal or/and pre-cardinal intra-criteria and inter-criteria information.
M-MACBETH provides another tool for this purpose, the \robustness analysis"
function. Let x and y be any two options. One would say that \x is globally more
attractive than y in the face of speci¯ed information" when k1v1ðxÞ þ k2v2ðxÞþk3v3ðxÞ > k1v1ðyÞ þ k2v2ðyÞ þ k3v3ðyÞ, for all single-value scales v1; v2; v3 and all
weights k1; k2; k3 compatible with the speci¯ed information (ordinal, pre-cardinal, or
cardinal). Such comparisons can be made by calculating, for each pair ðx; yÞ ofoptions, the minimal mðx; yÞ and maximum Mðx; yÞ values of the di®erence between½k1v1ðxÞ þ k2v2ðxÞ þ k3v3ðxÞ� and ½k1v1ðyÞ þ k2v2ðyÞ þ k3v3ðyÞ� for the speci¯ed
information. Three situations can arise: if mðx; yÞ � 0 and Mðx; yÞ > 0, then x is
globally more attractive than y (for the speci¯ed information); if mðx; yÞ < 0 and
M ðx; yÞ � 0, then y is globally more attractive than x; and, if mðx; yÞ < 0 and
M ðx; yÞ > 0, then neither of the two options is more attractive than the other. The
¯rst two are \additive dominance" situations and the third is \incomparability", as
de¯ned by Bana e Costa and Vincke (see Ref. 30).
Let us analyze whether the most attractive tender of the SOA case (tender C)
remains the same if one takes into account, simultaneously: (a) only the pre-cardinal
information elicited in Cr1 and Cr2 (i.e., the respective consistent sets of MACBETH
judgements of di®erence in attractiveness), (b) the cardinal information on Cr3 (i.e.,
the value function on Price) that lead to the ¯xed value scores v3ðAÞ ¼ 111; v3ðBÞ ¼101:5, and v3ðC Þ ¼ 27, and (c) the cardinal weighting information (i.e., the con-
sistent set of weighting judgements) that lead to the ¯xed weights k1 ¼ 0:55;
k2 ¼ 0:25, and k3 ¼ 0:20. The robustness analysis table (Table 3) shows that when
the value information of (a), (b), and (c) is selected simultaneously, tender C always
remains globally more attractive than tenders A and B (C additively dominates A
and B), although neither of the two tenders A and B can be robustly considered more
attractive than the other (A and B are incomparable).
The meaning of the contents of Table 3 in Fig. 12 is:
. C additively dominates A and B because, for all value scales v1 and v2 compatible
with the pre-cardinal information on criteria Cr1 and Cr2, respectively,
0:55v1ðCÞ þ 0:25v2ðCÞ þ 0:20� 27 > 0:55v1ðAÞ þ 0:25v2ðAÞ þ 0:20� 111 and
0:55v1ðCÞ þ 0:25v2ðCÞ þ 0:20� 27 > 0:55v1ðBÞ þ 0:25v2ðBÞ þ 0:20� 101:5;
Table 3. Table of robustness analysis
ð� denotes additive dominance and ?denotes incomparability).
! C B A
C ¼ � �B ¼ ?
A ¼
MACBETH 379
. A and B are incomparable because there exist at least two value scales v1 and u1and two value scales v2 and u2 compatible with the pre-cardinal information on Cr1and Cr2, respectively, for which,
0:55v1ðAÞ þ 0:25v2ðAÞ þ 0:20� 111 < 0:55v1ðBÞ þ 0:25v2ðBÞ þ 0:20� 101:5 and
0:55u1ðAÞ þ 0:25u2ðAÞ þ 0:20� 111 > 0:55u1ðBÞ þ 0:25u2ðBÞ þ 0:20� 101:5:
For example, if v1ðAÞ ¼ 33; v1ðBÞ ¼ 33; v2ðAÞ ¼ 40 and v2ðBÞ ¼ 60, we obtain
0:55v1ðAÞ þ 0:25v2ðAÞ þ 0:20� 111 ¼ 50:35 and
0:55v1ðBÞ þ 0:25v2ðBÞ þ 0:20� 101:5 ¼ 53:45:
If, however, u1ðAÞ ¼ 33; u1ðBÞ ¼ 33; u2ðAÞ ¼ 40; and u2ðBÞ ¼ 45, we obtain
0:55u1ðAÞ þ 0:25u2ðAÞ þ 0:20� 111 ¼ 50:35 and
0:55u1ðBÞ þ 0:25u2ðBÞ þ 0:20� 101:5 ¼ 49:70:
4. Some Concluding Re°ections
The MACBETH non-numerical approach to cardinal value measurement is based on
sound theory, like the traditional numerical approaches, but has the additional
advantage of o®ering a practical method of interactive veri¯cation of the reliability of
the preference information elicited. The MACBETH value-elicitation procedure is
composed of an input stage, aiming at eliciting, from an individual or group eva-
luator, a consistent set of non-numerical pairwise comparison judgements of quali-
tative di®erence in attractiveness, and an output stage, aiming at constructing from
the sets of judgements a multicriteria evaluation model that numerically measures
the relative attractiveness of options for the evaluator who made the judgements.
Respect for these judgements is the foundation stone of MACBETH, as modeled by
the condition of order preservation (COP). The nonconformity with COP-type
conditions of the eigenvalue procedure used to derive priorities from (ratio) pairwise
comparisons is viewed, from our constructive perspective of decision-aiding, as a
fundamental criticism (see Ref. 29) to the AHP method proposed by Saaty,84
although we recognize the in°uence of Saaty's original ideas (see, for example,
Ref. 85).
There are many well-known mistakes in making value trade-o®s (see
Refs. 60�62). The direct comparison of the criteria58 in terms of the intuitive notion
of importance can give rise to arbitrary recommendations from misleading average
sums of scores produced to aggregate value scores of options on the criteria. The
additive value model continues to be \by far the easiest to use and most familiar
model for such aggregations" (Ref. 48, p. 314) and it is therefore essential to avoid
the use of elicitation procedures that do not conform to the principles of additive
value measurement (for details, see the section title \the interpretation of criteria
weights" in Ref. 31, p. 147). For example, Edwards and Barron48 recognized the
weakness of the weighting procedure of SMART47 ��� which \ignores the fact that
range as well as importance must be re°ected in any weight" (Ref. 48, p. 316) ��� and
380 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
they replaced it with the correct swing-weighting technique. It is however a
numerical elicitation of weights and we were therefore motivated to propose the non-
numerical MACBETH weighting that can be viewed, appropriately, as a qualitative
swing-weighting technique.
A wide variety of real-world cases using the MACBETH approach and software,
both in the private and public sectors, are reported in the scienti¯c literature (see
Table 4; see also Ref. 12). In some of these multicriteria evaluation processes, we have
ourselves acted as decision-analysts and facilitators helping people with di®erent
tasks: managers, experts, civil servants, o±cers, politicians, etc. We observed that
they particularly liked three distinctive features:
. MACBETH's consistency checking at the input stage, especially the suggestions of
what to change if needed.
. The ability to put more than one qualitative rating in each cell of the matrix of
judgements, particularly useful when there are many disagreements within a
group; yet when the members of the group see the scale generated by MACBETH,
they usually agree that it is acceptable.
. The display of the range of possible scores on the scale so the ¯nal scale can be
adjusted.
As observed by Phillips (Ref. 78, p. 89): \Words are essential, more essential than
numbers, but a blending of the two can enable individuals and groups to achieve new
Table 4. Selected applications of MACBETH.
Agriculture, Manufacturing & Services:
Finance: Refs. 4, 18, 24 and 53
Performance measurement: Refs. 32�35, 42, 65�67 and 69
Production & service planning: Refs. 14, 73, 81, 82 and 86Quality management: Refs. 7, 39 and 40
Risk management: Refs. 43 and 54
Strategy & resource allocation: Refs. 10 and 15Supply chain management: Ref. 77
Energy:
Project prioritization and selection: Ref. 17Technology choice: Refs. 38, 52 and 72
Environment:
Landscape management: Ref. 89
Risk management: Ref. 22, 44 and 59Water resource management: Ref. 6
Medical: Refs. 45, 46, 74 and 75
Public Sector:
Con°ict analysis and management: Ref. 3, 23, 36, 51 and 83
Procurement: Refs. 9, 17 and 77
Project prioritization & resource allocation: Refs. 16, 21, 26, 27, 70, 76 and 87Strategic planning & development: Ref. 11
Human resource management & job selection: Refs. 8, 19, 20, 49, 55 and 56
MACBETH 381
depths of understanding which would not have been possible using either words or
numbers alone."
Concerning the extent to which the MACBETH approach has been helpful to
ANACOM, we reproduce here a message from Prof. Eduardo Cardadeiro, a member
of the Board of Directors, dated 2 November 2011: \In fact, some years ago ANA-
COM has adopted the use of Macbeth in its internal procedures for procurement of
goods and services over 75 thousand euros. Based on our experience this approach
has been very useful in several phases of those processes: (1) it helps to design
consistent valuation/selection criteria, ex-ante; (2) it makes the valuation of
alternatives much more objective and robust; and (3) the robustness analysis is
always performed before taking the ¯nal decision and, ex-post, helps to support the
decision, in case of litigation."
Acknowledgments
The authors would like to thank ANACOM for the opportunity to develop the SOA
case study and acknowledge the support provided by the Fundação para a Ciencia e aTecnologia.
References
1. K. Arrow, Social Choice and Individual Values (Wiley, New York, 1951).2. Bana Consulting, M-MACBETH Version 1.1: User Manual (Bana Consulting, Lisbon,
2005), http://www.m-macbeth.com/help/pdf/M-MACBETH%20User's%20Guide.pdf.3. C. A. Bana e Costa, The use of multi-criteria decision analysis to support the search for
less con°icting policy options in a multi-actor context: Case study, Journal of Multi-Criteria Decision Analysis 10(2) (2001) 111�125.
4. C. A. Bana e Costa, L. A. Barroso and J. O. Soares, Qualitative modeling of creditscoring: A case study in banking, European Research Studies Journal 5(1�2) (2002)37�51.
5. C. A. Bana e Costa and E. Beinat, Model-structuring in public decision-aiding (LSEOR05.79) (London School of Economics and Political Science, London, 2005), http://eprints.lse.ac.uk/22716/1/05079.pdf.
6. C. A. Bana e Costa, P. A. da Silva and F. N. Correia, Multicriteria evaluation of °oodcontrol measures: The case of Ribeira do Livramento, Water Resources Management18(3) (2004) 263�283.
7. C. A. Bana e Costa, M. Carnero and M. D. Oliveira, A multi-criteria model for auditing aPredictive Maintenance Programme, European Journal of Operational Research 217(2)(2012) 381�393.
8. C. A. Bana e Costa and M. P. Chagas, A career choice problem: An example of how to useMACBETH to build a quantitative value model based on qualitative value judgments,European Journal of Operational Research 153(2) (2004) 323�331.
9. C. A. Bana e Costa, E. C. Correa, J. M. De Corte and J.-C. Vansnick, Facilitating bidevaluation in public call for tenders: A socio-technical approach, Omega-InternationalJournal of Management Science 30(3) (2002) 227�242.
10. C. A. Bana e Costa, E. Correa, L. Ensslin and J.-C. Vansnick, Decision support systems inaction: Integrated application in a multicriteria decision aid process, European Journal ofOperational Research 113(2) (1999) 315�335.
382 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
11. C. A. Bana e Costa, M. L. Costa-Lobo, I. A. J. Ramos and J.-C. Vansnick, Multicriteriaapproach for strategic town planning: The case of Barcelos, in Aiding Decisions withMultiple Criteria: Essays in Honor of Bernard Roy, eds. D. Bouyssou, E. Jacquet-Lagr�eze, P. Perny et al. (Kluwer Academic Publishers, Boston, 2002), pp. 429�456.
12. C. A. Bana e Costa, J. M. De Corte and J. C. Vansnick, On the mathematical foundationsof MACBETH, in Multiple Criteria Decision Analysis: The State of the Art Surveys, eds.J. Figueira, S. Greco and M. Ehrgott (Springer, New York, 2005), pp. 409�442.
13. C. A. Bana e Costa, J. M. De Corte and J. C. Vansnick, MACBETH (MeasuringAttractiveness by a Categorical Based Evaluation Technique), in Wiley Encyclopedia inOperational Research and Management Science, ed. J. J. Cochran 4 (Wiley, New York,2011), pp. 2945�2950.
14. C. A. Bana e Costa, L. Ensslin and A. P. Costa, Structuring the process of choosing ricevarieties at the south of brazil, multicriteria analysis for land-use management, inEnvironment and Management, eds. E. Beinat and P. Nijkamp 9 (Kluwer AcademicPublishers, Dordrecht, 1998), pp. 33�45.
15. C. A. Bana e Costa, L. Ensslin and I. J. Zanella, A real-world MCDA application incellular telephony systems, in Trends in Multicriteria Decision Making, eds. T. J. Stewartand R. C. van den Honert (Springer, Berlin, 1998), pp. 412�423.
16. C. A. Bana e Costa, T. G. Fernandes and P. V. D. Correia (2006), Prioritisation of publicinvestments in social infrastructures using multicriteria value analysis and decision con-ferencing: A case study, International Transactions in Operational Research 13(4) (2006)279�297.
17. C. A. Bana e Costa, J. C. Lourenço, M. P. Chagas and J. C. Bana e Costa, Developmentof reusable bid evaluation models for the Portuguese Electric Transmission Company,Decision Analysis 5(1) (2008) 22�42.
18. C. A. Bana e Costa, J. C. Lourenço and J. O. Soares, An interval weighting assignmentmodel for credit analysis, Journal of Financial Decision Making 3(2) (2007) 1�9.
19. C. A. Bana e Costa, P. A. F. Martins, M. D. Oliveira, A. Sernadas and C. A. Mota Soares,Faculty evaluation using multicriteria value measurement, Advances in Mathematicaland Computational Methods (WSEAS Press, 2010), pp. 287�290.
20. C. A. Bana e Costa and M. D. Oliveira, A multicriteria decision analysis model for facultyevaluation, Omega-International Journal of Management Science 40(4) (2012) 424�436.
21. C. A. Bana e Costa and R. C. Oliveira, Assigning priorities for maintenance, repair andrefurbishment in managing a municipal housing stock, European Journal of OperationalResearch 138(2) (2002) 380�391.
22. C. A. Bana e Costa, C. S. Oliveira and V. Vieira, Prioritization of bridges and tunnels inearthquake risk mitigation using multicriteria decision analysis: Application to Lisbon,Omega-International Journal of Management Science 36(3) (2008) 442�450.
23. C. A. Bana e Costa, F. N. Silva and J.-C. Vansnick, Con°ict dissolution in the publicsector: A case-study, European Journal of Operational Research 130(2) (2001) 388�401.
24. C. A. Bana e Costa and J. O. Soares, A multicriteria model for portfolio management,European Journal of Finance 10(3) (2004) 198�211.
25. C. A. Bana e Costa and J.-C. Vansnick, MACBETH ��� An interactive path towards theconstruction of cardinal value functions, International Transactions in OperationalResearch 1(4) (1994) 489�500.
26. C. A. Bana e Costa and J.-C. Vansnick, Applications of the MACBETH approach in theframework of an additive aggregation model, Journal of Multi-Criteria Decision Analysis6(2) (1997) 107�114.
27. C. A. Bana e Costa and J.-C. Vansnick, The Macbeth approach: Basic ideas, software,and an application, in Advances in Decision Analysis, eds. N. Meskens and M. Roubens(Springer, Dordrecht, 1999), pp. 131�157.
MACBETH 383
28. C. A. Bana e Costa and J.-C. Vansnick, Cardinal value measurement with MACBETH,in Decision Making: Recent Developments and Worldwide Applications, eds. S. H.Zanakis, G. Doukidis and C. Zapounidis (Kluwer Academic Publishers, Dordrecht, 2000),pp. 317�329.
29. C. A. Bana e Costa and J. C. Vansnick, A critical analysis of the eigenvalue method usedto derive priorities in AHP, European Journal of Operational Research 187(3) (2008)1422�1428.
30. C. A. Bana e Costa and P. Vincke, Measuring credibility of compensatory preferencestatements when trade-o®s are interval determined, Theory and Decision 39(2) (1995)127�155.
31. V. Belton and T. J. Stewart, Multiple Criteria Decision Analysis: An IntegratedApproach (Springer, Berlin, 2002).
32. L. Berrah and V. Clivill�e, Towards an aggregation performance measurement systemmodel in a supply chain context, Computers in Industry 58(7) (2007) 709�719.
33. L. Berrah, G. Mauris and J. Montmain, Monitoring the improvement of an overallindustrial performance based on a Choquet integral aggregation, Omega-InternationalJournal of Management Science 36 (2008) 340�351.
34. L. Berrah, G. Mauris, J. Montmain and V. Cliville, E±cacy and e±ciency indexes for amulti-criteria industrial performance synthesized by Choquet integral aggregation,International Journal of Computer Integrated Manufacturing 21(4) (2008) 415�425.
35. L. Berrah, G. Mauris and F. Vernadat, Industrial performance measurement: Anapproach based on the aggregation of unipolar or bipolar expressions, InternationalJournal of Production Research 44(18�19) (2006) 4145�4158.
36. D. Bollinger and J. Pictet, Potential use of e-democracy in MCDA processes. Analysis onthe Basis of a Swiss case, Journal of Multi-Criteria Decision Analysis 12 (2003) 65�76.
37. T. B€orgers and C. Dustman, Awarding telecom licenses: The recent European experience,Economic Policy 36 (2003) 215�268.
38. J. Burton and K. Hubacek, Is small beautiful? A multicriteria assessment of small-scaleenergy technology applications in local governments, Energy Policy 35 (2007)6402�6412.
39. M. C. Carnero, Evaluating a maintenance department in a service company, Inter-national Journal of Mathematical Models and Methods in Applied Sciences 3(3) (2009)230�237.
40. M. C. Carnero, Maintenance audit by means of an additive multicriteria model, inAdvances in Marketing, Management and Finances, eds. S. Hashemi and C. Vobach(WSEAS Press, 2009), pp. 147�152.
41. J. F. Chang, Business Process Management Systems: Strategy and Implementation(Auerbach Publications, Boca Raton, 2006).
42. V. Cliville, L. Berrah and G. Mauris, Quantitative expression and aggregation of per-formance measurements based on the MACBETH multi-criteria method, InternationalJournal of Production Economics 105(1) (2007) 171�189.
43. H. R. Costa, M. D. O. Barros and G. Travassos, Evaluating software project portfoliorisks, Journal of Systems & Software 80 (2007) 16�31.
44. F. Dall'Osso, M. Gonella, G. Gabbianelli, G. Withycombe and D. Dominey-Howes, Arevised (PTVA) model for assessing the vulnerability of buildings to tsunami damage,Natural Hazards and Earth System Sciences 9(5) (2009) 1557�1565.
45. A. K. A. de Castro, P. R. Pinheiro and M. Pinheiro, Towards the neuropsychologicaldiagnosis of Alzheimer's disease: A hybrid model in decision making, in Best Practices forthe Knowledge Society: Knowledge, Learning, Development and Technology for All, eds.M. D. Lytras et al. (Springer, Berlin, 2009), pp. 522�531.
384 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
46. A. K. A. de Castro, P. R. Pinheiro, M. Pinheiro and I. Tamanini, Towards the appliedhybrid model in decision making: A neuropsychological diagnosis of Alzheimer's diseasestudy case, International Journal of Computational Intelligence Systems 4(1) (2011)89�99.
47. W. Edwards, How to use multiattribute utility. measurement for social decisionmaking,IEEE Transactions on Systems, Man and Cybernetics 7 (1977) 326�340.
48. W. Edwards and F. H. Barron, SMARTS and SMARTER: Improved simple methods formultiattribute utility measurement, Organizational Behavior and Human DecisionProcesses 60 (1994) 306�325.
49. L. Ensslin, A. Dutra and S. R. Ensslin, MCDA: A constructivist approach to the man-agement of human resources at a governmental agency, International Transactions onOperational Research 7(1) (2000) 79�100.
50. B. Fasolo and C. A. Bana e Costa, Tailoring value elicitation to decision makers'numeracy and °uency: Expressing value judgements in numbers or words (London Schoolof Economics and Political Science, London, 2011).
51. P. Eklund, A. Rusinowska and H. D. Swart, A consensus model of political decision-making, Annals of Operations Research 158 (2008) 5�20.
52. M. B. Fernandes, M. C. Almeida and A. G. Henriques, Assessing desalination as a sus-tainable alternative using a multiple criteria decision support model, Water Practice &Technology 3(3), doi:10.2166/wpt.2008.078
53. F. A. F. Ferreira, S. P. Santos and P. M. M. Rodrigues, Adding value to bank branchperformance evaluation using cognitive maps and MCDA: A case study, Journal of theOperational Research Society 62(7) (2010) 1320�1333.
54. M. S. M. Figueiredo and M. D. Oliveira, Prioritizing risks based on multicriteria decisionaid methodology: Development of methods applied to ALSTOM Power, IEEE Int.Conf. Industrial Engineering and Engineering Management (Hong Kong, China, 2009),pp. 1568�1572.
55. A. B. Filho, A. S. Marçal, G. Costa, P. R. Pinheiro and R. F. Pinheiro, A novel approachbased on sta® scheduling optimization in information technology projects, InternationalJournal of Computer Science and Network Security 9(9) (2009) 277�286.
56. T. Gurbuz, Multiple criteria human performance evaluation using Choquet integral,International Journal of Computational Intelligence Systems 3(3) (2010) 290�300.
57. Z. Hatush and M. Skitmore, Criteria for contractor selection, Construction ManagementEconomics 15(1) (1997) 19�38.
58. D. P. Henriksen and S. W. Palocsay, An excel-based decision support system for scoringand ranking proposed R&D projects, International Journal of Information Technology &Decision Making 7(3) (2009) 529�546.
59. F. Joerin, G. Cool, M. J. Rodriguez, M. Gignac and C. Bouchard, Using multi-criteriadecision analysis to assess the vulnerability of drinking water utilities, EnvironmentalMonitoring and Assessment 166(1�4) (2010) 313�330.
60. R. L. Keeney, Value-Focused Thinking: A Path to Creative Decisionmaking (HarvardUniversity Press, Cambridge, MA, 1992).
61. R. L. Keeney, Common mistakes in making value trade-o®s, Operations Research 50(6)(2002) 935�945.
62. R. L. Keeney, Eliciting knowledge about values for public policy decisions, InternationalJournal of Information Technology & Decision Making 5(4) (2006) 739�749.
63. R. L. Keeney and H. Rai®a, Decision with Multiple Objectives: Preferences and ValueTradeo®s (Cambridge University Press, New York, 1993).
64. C. W. Kirkwood, Strategic Decision Making: Multiobjective Decision Analysis withSpreadsheets (Duxbury Press, Belmont, CA, 1997).
MACBETH 385
65. R. T. O. Lacerda, L. Ensslin and S. R. Ensslin, A performance measurement framework inportfolio management: A constructivist case, Management Decision 49(4) (2011)648�668.
66. R. T. O. Lacerda, L. Ensslin and S. R. Ensslin, A performance measurement view of ITproject management, International Journal of Productivity and Performance Manage-ment 60(2) (2011) 132�151
67. M. Lauras, G. Marques and D. Gourc, Towards a multi-dimensional project PerformanceMeasurement System, Decision Support Systems 48 (2010) 342�353.
68. E. A. Marks and M. Bell, Executive's Guide to Service-Oriented Architecture (JohnWiley& Sons, Hoboken, 2006).
69. G. Marques, D. Gourc and M. Lauras, Multi-criteria performance analysis for decisionmaking in project management, International Journal of Production Management 29(8)(2010) 1057�1069.
70. R. Mateus, J. A. Ferreira and J. Carreira, Multicriteria decision analysis (MCDA):Central Porto high-speed railway station, European Journal of Operational Research187(1) (2008) 1�18.
71. D. L. McLain and R. J. Aldag, Complexity and familiarity with computer assistance whenmaking ill-structured business decisions, International Journal of Productivity and Per-formance Management 8(3) (2009) 407�426.
72. F. Montignac, I. Noirot and S. Chaurdoune, Multi-criteria evaluation of on-boardhydrogen storage technologies using the MACBETH approach, International Journal ofHydrogen Energy 34 (2009) 4561�4568.
73. J. Montmain, C. Sanchez and M. Vinches, Multi criteria analyzes for managing motorwaycompany facilities: The decision support system SINERGIE, Advanced EngineeringInformatics 23(3) (2009) 265�287.
74. L. Moraes, R. Garcia, L. Ensslin and M. J. Conceição, The multicriteria analysis forconstruction of benchmarkers to support the clinical engineering in the healthcaretechnology management, European Journal of Operational Research 200(3) (2010)607�615.
75. L. C. Nunes, P. R. Pinheiro and T. C. Pequeno, Toward an application to psychologicaldisorders diagnosis, Software Tools and Algorithms for Biological Systems 696 (2011)573�580.
76. M. D. Oliveira, T. C. Rodrigues, C. A. Bana e Costa and A. B. de S�a, Prioritizing healthcare interventions: A multicriteria resource allocation model to inform the choice ofcommunity care programmes, in Advanced Decision Making Methods Applied to HealthCare, eds. E. Tanfani and A. Testi (Springer, forthcoming), pp. 139�152.
77. R. C. Oliveira and J. C. Lourenço, A multicriteria model for assigning new orders toservice suppliers, European Journal of Operational Research 139 (2002) 390�399.
78. L. D. Phillips, Decision Analysis in the 1990s, in Tutorial Papers in Operational Research,eds. A. Shahani and R. Stainton (The Operational Research Society, Birmingham, 1989),pp. 73�90.
79. L. D. Phillips, Decision conferencing, in Advances in Decision Analysis: From Foun-dations to Applications, eds. W. Edwards, R. F. Miles and D. vonWinterfeldt (CambridgeUniversity Press, New York, 2007), pp. 375�399.
80. L. D. Phillips and C. A. Bana e Costa (2007), Transparent prioritisation, budgeting andresource allocation with multi-criteria decision analysis and decision conferencing, Annalsof Operations Research 154(1) 51�68.
81. P. R. Pinheiro, A. K. A. de Castro and M. C. D. Pinheiro, Application multicriteriadecision analysis on TV digital, in Advances in Computer and Information Sciences andEngineering, ed. T. Sobh (Springer, USA, 2008), pp. 39�44.
386 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick
82. A. Rodrigues, P. R. Pinheiro, M. M. Rodrigues, A. B. Albuquerque and F. M. Gonçalves,Towards the selection of testable use cases and a real experience, in Best Practices for theKnowledge Society: Knowledge, Learning, Development and Technology for All, ed. M. T.Lytras (Springer, Berlin, 2009), pp. 513�521.
83. M. Roubens, A. Rusinowska and H. de Swart, Using MACBETH to determine utilities ofgovernments to parties in coalition formation, European Journal of Operational Research172(2) (2006) 588�603.
84. T. L. Saaty, A scaling method for priorities in hierarchical structures, Journal of Math-ematical Psychology 15(3) (1997) 234�281.
85. T. L. Saaty and S. Mujgan, Extending the measurement of tangibles to intangibles,International Journal of Information Technology & Decision Making 8(1) (2009) (7�27).
86. C. Sanchez, J. Montmain, M. Vinches and B. Mahieu, Planning of maintenance oper-ations for a motorway operator based upon multicriteria evaluations over a ¯nite scaleand sensitivity analyzes, in Informatics in Control, Automation and Robotics, eds. A. J.Cetto, J. L. Ferrier, J. M. C. D. and J. Filipe (Springer, Berlin, 2008), pp. 23�35.
87. R. Sanchez-Lopez, C. A. Bana e Costa and B. De Baets, The MACBETH approach formulti-criteria evaluation of development projects on cross-cutting issues, Annals ofOperations Research (2011), doi:10.1007/s10479-011-0877-4.
88. E. H. Schein, Process Consultation Revisited: Building the Helping Relationship (AddisonWesley Longman, MA, 1998).
89. N. Soguel, M.-J. Martin and A. Tangerini, The impact of housing market segmentationbetween tourists and residents on the hedonic price for landscape quality, Swiss Journal ofEconomics and Statistics 144(4) (2008) 655�678.
90. D. von Winterfeldt and W. Edwards, Decision Analysis and Behavioral Research(Cambridge University Press, New York, 1986).
MACBETH 387