Diversity of ability and cognitive style for group decision processes

Information Sciences 179 (2009) 542–558

Contents lists available at ScienceDirect

Information Sciences

journal homepage: www.elsevier .com/locate / ins

Diversity of ability and cognitive style for group decision processes

David West a,*, Scott Dellana b

a Department of Marketing and Supply Chain Management, College of Business Administration, East Carolina University, 3205 Bate Building,Greenville, NC 27858-4353, USAb Department of Marketing and Supply Chain Management, College of Business Administration, East Carolina University, Greenville, NC 27858-4353, USA

a r t i c l e i n f o

Article history:Received 4 May 2007Received in revised form 24 October 2008Accepted 27 October 2008

Keywords:Multi-agent group decisionsAbility diversityCognitive diversityBankruptcy detection

0020-0255/$ - see front matter � 2008 Elsevier Incdoi:10.1016/j.ins.2008.10.028

* Corresponding author. Tel.: +1 252 328 6370; faE-mail addresses: [email protected] (D. West), dell

a b s t r a c t

This research investigates the potential for two forms of error diversity (ability diversityand diversity of cognitive style) to increase the accuracy of multi-agent group decision pro-cesses. An experimental methodology is employed that rigorously controls for the sourcesof error diversity. The results indicate that ability diversity decreases group decision errorsby approximately 4%. Cognitive diversity is much more effective; decision errors arereduced by approximately 13% by groups formed from four cognitive classes. As sourcesof ability and cognitive diversity increase, the generalization error of the group decisiondecreases, and the prominence of the most capable member (i.e., expert) in the groupdiminishes. Thus, the popular reliance on using more capable members to create high per-formance homogenous groups may be misguided. This research indicates that a betterstrategy is to create groups of members that ‘think differently’ and cooperate to producea group decision. Using this strategy, we are able to reduce the group decision error intwo bankruptcy detection data sets by 11–47%. Reductions of this magnitude in high vol-ume, high value, and repetitive decision environments characterizing the financial domainare extremely significant, where error reductions of even a fraction of a percent arewelcome.

� 2008 Elsevier Inc. All rights reserved.

1. Introduction

The concepts of multi-agent decision systems [53] and cooperative intelligent systems [45] have recently been proposedfor decision applications in economics and finance [11]. The literature distinguishes between multi-agent systems that em-ploy competitive synthesis and systems that use cooperative synthesis [45]. In a competitive synthesis system, group mem-bers work separately on the same problem and the best individual solution is identified as the group’s solution. Logically, theaccuracy of competitive synthesis systems can be increased by identifying and including smarter experts. By contrast, incooperative synthesis systems, the individual capabilities of all group members are aggregated to produce a group solution.The primary focus to increase the accuracy of a cooperative synthesis group is to create error diversity among the groupmembers so that group members think differently and disagree on some aspects of the problem theory. There is no advan-tage in cooperative synthesis systems if all group members have similar problem solving mechanisms and come to the samedecision.

Several studies conclude that a problem’s error can be reduced by a cooperative synthesis group decision processes(sometimes referred to as ensembles) [21,22,32]. Despite the recent evidence of the superiority of cooperative synthesisgroup decisions, most of the research to date in the financial domain focuses on competitive synthesis, finding smarterexperts. The study of the impact of a group’s error diversity on the accuracy of cooperative synthesis is an underdeveloped

. All rights reserved.

x: +1 252 328 [email protected] (S. Dellana).

mailto:[email protected]

mailto:[email protected]

http://www.sciencedirect.com/science/journal/00200255

http://www.elsevier.com/locate/ins

D. West, S. Dellana / Information Sciences 179 (2009) 542–558 543

research area. While there has been substantial research on data manipulation methods to create group error diversity, otherdiversity sources such as variations in group member’s capabilities or cognitive styles are largely ignored [10].

To the best of our knowledge, this is the first study to systematically evaluate the importance of ability diversity and cog-nitive diversity in group decision processes. Further, we choose to focus on the financial application of bankruptcy detectionbecause this decision environment is characterized by high volume (there are tens of thousands of firms to monitor), highsignificance (there are millions of dollars at stake), and repetitive activity (requires constant updating and monitoring). Ourmotivations for this research are supported by the following quote from Brown et al. [10]. ‘‘The number of investigations intousing different types of neural networks (or different types of learners in general) in ensembles is disappointingly small. If we wantdiverse errors in our ensembles, it makes sense that using different types of function approximator may produce this.”

2. Literature review

The literature search focuses first on the current theoretical knowledge of multi-agent cooperative synthesis systems andthe role of error diversity. We then describe a limited set of the research conducted on competitive synthesis systems, thesearch for smarter experts. The final subsection characterizes the research on error diversity mechanism for cooperative syn-thesis systems, the search for smarter groups.

2.1. Theory of cooperative synthesis

The theoretical evaluation of the relationship between sources of diversity and group decision accuracy begins with thequadratic error of the group decision and attempts to decompose the quadratic error into terms that measure both the aver-age quadratic error of the individual group members and the group diversity effect. The group classification decision involvesalgorithms that output discrete class labels, zero–one loss functions, and aggregate decisions produced by majority vote.Unfortunately, the theoretical analysis for classification decisions is complex and is still being developed. We will brieflyhighlight the theoretical development for regression group decisions that are characterized by real-valued outputs with con-vex combinations. This problem has been easier to analyze and is well developed. Those readers interested in more detail ofthe chronological development of this theory are referred to Brown et al. [10].

For regression, the group decision Dg, is a convex combination where each individual member’s decision di is weighted bywiDg ¼

Piwidi where wi > 0 and

Piwi ¼ 1. Krogh and Vedelsby [34] algebraically rearranged the quadratic error of the group

regression decision at a test value, v, to produce the following form, referred to as the ambiguity decomposition:

ðDg � vÞ2 ¼X

i

wiðdi � vÞ2 �X

i

wiðdi � DgÞ2: ð1Þ

The first term on the right-hand side of Eq. (1) expresses the group error as a function of the weighted average error of theindividual group members. The second term, referred to as the ambiguity term, is always a positive value and measures theweighted average variability among the group members. Larger disagreements among group members create larger ambi-guity values, which in turn decrease the group error. Minimizing the group error involves a complex trade-off: larger ambi-guity terms are associated with higher individual error rates. Therefore, higher levels of diversity do not guarantee moreaccurate decisions. It is essential to achieve an effective balance between diversity (the ambiguity term) and individual accu-racy (the average error term) to minimize the group error. We can also observe from Eq. (1) that the error of the convex-combined group will be less than or equal to the average error of the individuals.

For group classification decisions we depend on the qualitative advice of Hansen and Salamon [21] who claim that a nec-essary and sufficient condition for a majority voting group to be more accurate than its individual members is that the indi-viduals are accurate and diverse.

2.2. Competitive synthesis: the search for smarter experts

There has been a significant research effort devoted to competitive synthesis systems. This methodology usually takes theform of competitive tournaments with a new champion competing against a subset of traditional algorithms. In this section,we review some of the competitive synthesis examples but caution the reader that this is not an exhaustive coverage. One ofthe first efforts that Wilson and Sharda [63] who compare the predictive capabilities for firm bankruptcy of neural networksand multivariate discriminant analysis; they conclude that neural networks predict firm bankruptcies significantly betterthan discriminant analysis. Tam and Kiang [56] investigate a neural network to identify bank defaults. They compare in acompetitive manner a neural network, linear discriminant analysis, logistic regression, k-nearest neighbor, and recursivepartitioning trees. Their results demonstrate that the neural network shows promise for evaluating bank default in termsof predictive accuracy, adaptability, and robustness. Zhang et al. [69] also conclude that neural networks are significantlybetter than logistic regression in bankruptcy prediction.

Serrano-Cinca [52] propose Self-Organizing Feature Maps for financial diagnosis; the maps project financial ratio infor-mation from a high dimensional space to a two dimensional map. The Self-Organizing Feature Map is compared with lineardiscriminant analysis as well as with multilayer perceptron neural network models [52]. Tsakonas et al. [57] demonstrate

544 D. West, S. Dellana / Information Sciences 179 (2009) 542–558

the efficient use of hybrid intelligent systems based on genetic programming for solving the bankruptcy detection problem.Huang et al. [27] introduce support vector machines to the bankruptcy detection problem in an attempt to provide a modelwith better explanatory power. The authors obtain prediction accuracy of 80% for both the multilayer perceptron neuralnetwork and support vector machines using data from the United States and Taiwan markets [27]. Lee et al. [37] developeda hybrid neural network for bankruptcy prediction and conclude from Korean bankruptcy data that the hybrid neuralnetwork is more promising for bankruptcy prediction in terms of accuracy and adaptability. Finally, a number of otherauthors have focus on variations of neural network models including the following [3,6,9,13,15,16,20,25,28,35,36,45–49,62,66,67,69,74,75].

2.3. Cooperative synthesis: the search for smarter groups

The ability of competitive synthesis systems to produce accurate decisions is dependent on a number of formal mecha-nisms that generate diversity of error among group members [10,71]. A substantial set of research has accumulated on diver-sity generating mechanisms. We will limit our discussion to some of the most significant publications and to those that mostclosely relate to our research. The reader is advised to consult Brown et al. [10] for a comprehensive survey of error diversityand diversity creation methods.

Brown et al. [10] categorizes diversity creation methods as follows: manipulation of training data, manipulation of archi-tectures, and hypothesis space traveled. We refer to diversity sources created by the manipulation of training data as contentdiversity. The category of error diversity that Brown et al. [10] refers to as manipulation of architecture includes ability diver-sity and diversity of cognitive style. Ability diversity is created by manipulating the parameters of a single architecture tocreate learners with differing capacities. Diversity of cognitive style (which we refer to hereafter as simply cognitive diversity)results from manipulation of architectures to create agents that think differently.

The initial empirical research on cooperative synthesis systems relies primarily on content diversity (data manipulationmethods). Two of the most common methods to generate content error diversity are bagging [7] and boosting [58]. Baggingand boosting operate directly on the examples (i.e., the rows) in the training set. Bagging implicitly creates data manipula-tion error diversity by random sampling with replacement to form bootstrap replicates, while boosting increases the propor-tion of observations that are not correctly classified during prior training cycles. Content error diversity can also be createdby variable subset selection (randomly choosing subsets of feature variables), adding noise to the input or target data, and byrandomly reversing a proportion of the output targets in the training data.

Bootstrap aggregation (bagging) is a popular way of achieving content error diversity; each member learns from a uniquebootstrap replicate of the original set of learning examples [7]. Breiman [7] demonstrates this by testing bootstrap groups ofclassification and regression trees on several benchmark data sets, reporting reductions in decision error ranging from 6% to77%. A number of other studies focus primarily on decision agent systems with content error diversity[5,14,23,26,39,55,60,68,72–74]. Generally, these studies employ a single cognitive style (algorithm) and a single ability level.Skurichina and Duin [54] investigate bagging for linear discriminant classifiers. Franke [19], and West [62] research boot-strapping for groups neural networks while Kim [30] uses both bagging and boosting for groups of support vector machinesin multi-class classification problems. Bauer and Kohavi [4] and Opitz and Maclin [42] also research forms of bagging andboosting on decision trees and naïve Bayes algorithms. The explicit content error diversity generated by boosting is studiedby Drucker et al. [18] and Schwenk and Bengio [50] for homogenous groups of neural networks and by Kearns and Mansour[31] for decision trees.

In contrast to bagging and boosting, subset feature selection creates content error diversity by operating on the featurevariables (the columns of the training set). Subset feature selection (selecting random subsets of feature variables for eachlearner) is particularly useful for problems with large feature sets such as DNA micro-array. Abdel-Aal [1] uses abductivenetwork groups trained on different features for improving classification accuracy in medical diagnosis. Kim and Cho [29]research subset selection in groups learning DNA micro-array, while Kim [30] reports on variable subsets in groups of neuralnetworks to maximize direct mailing response. Yu and Cho [65] create feature subset selection for groups of support vectormachines to analyze customers’ historical purchase data.

The practice of injecting noise in the learning data is a data manipulation method intended to increase content errordiversity by distributing group members over a more diverse set of local minima in the error surface creating a more inde-pendent set of estimators. Raviv and Intrator [47] add noise to boosted groups of neural networks, and Diettrich [17] inves-tigates randomization of input data for groups of decision trees and compares the results with bagging and boosting. Zhang[70] adds noise to input data for neural network groups that predict time series forecasts.

Content error diversity can be created by data manipulation methods that operate on the training output data. In thisresearch thread, Melville and Mooney [40] study diversity created by oppositional relabeling of artificial training examplesin decision trees, while Breiman [8] examines the potential to increase group prediction accuracy by perturbing the outputsalone.

A few studies have implicitly examined sources of error diversity generated by the manipulation of architecture(i.e., varying the ability level and/or the cognitive style of group members). Versace et al. [59] estimate the performanceof groups consisting of recurrent back-propagation and radial basis function neural networks for predicting the exchange-traded fund DIA. Mukkamala et al. [41] construct a group of neural networks, support vector machines, and multivariateadaptive regression splines for intrusion detection. Peddabachigari et al. [44] also study the problem of intrusion detection


with heterogeneous groups of decision trees and support vector machines. Cho [12] introduces a mechanism for adapting thestructure of a group of self-organizing maps. Malhotra et al. [38] create groups of neural networks, discriminant analysis,quadratic discriminant analysis, k-nearest neighbor, and multinomial logistic regression analysis in a retail department storedecision environment. Mangiameli et al. [39] and West et al. [61] both employ cognitive diversity to analyze the accuracy ofmedical decision support systems and breast cancer detection, respectively.

2.4. Contributions of this research

While the effectiveness of data manipulation to create content error diversity is extensively researched, there is littlerecognition in the literature of the impact of either ability diversity or cognitive diversity on group decision error. We areunable to find any research that explicitly controls by diversity mechanism and estimates the relative contribution togroup decision accuracy. The objective of this paper is to estimate the potential of ability diversity and cognitive diversitysources to reduce cooperative synthesis group decision error. We formulate the following specific research questions tosupport this objective.

1. What decision accuracies can be expected of groups that are limited to content diversity?2. Will the inclusion of ability diversity in the groups investigated in question 1 increase decision accuracy?3. Will the inclusion of cognitive diversity in the groups investigated in question 2 increase decision accuracy?4. How does the composition of group membership differ between the most accurate and least accurate groups?

In the next section, we discuss the sources of group diversity investigated in this research. Section 4 explains our researchmethodology and the experimental design used to estimate the group decision error and the effects for changing group com-position. The mean generalization error for several levels of diversity is presented in Section 5 for US bankruptcy data, and inSection 6 for Spanish bank bankruptcy data. The changing composition of groups is analyzed in Section 7. We conclude inSection 8 with a discussion of results and implications to guide the development of intelligent decision systems for financialapplications.

3. Controlling sources of error diversity

Before discussing error diversity, we need to clarify some terminology used in this paper. We adopt the expression ‘‘deci-sion agent” in our research to define an entity that has a specific cognitive style and a mechanism for learning from examplesof a problem domain in its environment. The decision agent has the ability to render decisions about the status of objects inits environment. This is a more limited concept of an agent, which is generally defined as an entity that has some form ofperception of its environment, can act, and can communicate with other agents. While it is technically possible for our deci-sion agents to communicate with each other during problem solving, we choose to have them work autonomously in thisresearch. We use a group of terms synonymously in the remainder of the paper. A decision group can also be identifiedas an ensemble; a group member is identified as a decision agent, as a member, or as an algorithm depending on the contextof the discussion.

In this research, we investigate three major sources of error diversity in group decision processes. The first source ofdiversity is diversity in learning content caused by the perturbation of the training sets of the bagging algorithm. Thesecond source of diversity is generated by the variation in the agents cognitive capacity as the agent’s architecture isvaried. For example, as the MLP hidden layer neurons vary from 2 to 10 (Appendix), the capacity of the MLP agent torepresent complex decision boundaries increases. This form of diversity, which we refer to as ability diversity, resultsfrom agents that think the same (i.e., have the same approach to solving the problem) but have more or less ability torepresent complex decision boundaries. The third source of diversity is created when group membership consists ofagents from different cognitive classes; we refer to this as cognitive diversity. For example, we will establish thatMLP and KNN agents think differently, i.e., they have different ways of representing the problem and use different heu-ristics to achieve improved solutions [24]. Each source of diversity will be discussed briefly in the followingsubsections.

3.1. Bagging to create content diversity

The bagging algorithm is the most common method of content diversity for multi-agent decision systems. It functions byvarying the learning content available to each intelligent agent. Each agent has some specific learning examples emphasizedthrough duplication in its bootstrap replicate, while other learning instances are missing. For unstable agents, these differ-ences in learning content result in significantly different parameter estimates and potentially a diversity of member deci-sions that create more accurate collective decisions.

Unique training sets (bootstrap replicates) are formed to estimate agent parameters for each group member by samplingwith replacement from the original training set partition Li. Details of the bagging algorithm are given below [7]. In step 4,the collective group decision is created by a majority vote of B hypothesis.


Algorithm for baggingGiven: training set of size n and base classification algorithm Ct(x)

Step 1. Input sequence of training samples (x1,y1), . . . , (xn,yn) with labels y 2 Y = (�1,1)Step 2. Initialize probability for each example in learning set, D1(i) = 1/n and set t = 1Step 3. Loop while t < B group members

(a) Form training set of size n by sampling with replacement from distribution Dt

(b) Get hypothesis ht : X ? Y(c) Set t = t + 1End of loop

Step 4. Output the final group hypothesis C�ðxiÞ ¼ hfinalðxiÞ ¼ arg maxPB

t¼1IðCtðxÞ ¼ yÞ

3.2. Agent architecture and ability diversity

Ability diversity is characterized as agents that think alike (i.e., are from the same cognitive class) but have different capa-bilities to form decision boundaries. Higher capacity agents can form more complex decision boundaries; lower capacityagents form simpler decision boundaries.

For agents from cognitive classes like k-nearest neighbor, Kernel density estimation, multilayer perceptron neural net-work and radial basis function neural network, it is possible to vary the capacity (and therefore the ability) of the agentby the choice of agent architecture. For example, MLPa,MLPb, . . . ,MLPe form a sequence of MLP architectures with increasingnumbers of hidden neurons and therefore increasing capacities (see Appendix). The term ‘capacity’ refers to a concept fromstatistical learning theory that refers to the representational abilities of the agents [58]. High capacity agents can form morecomplex decision boundaries and consequently, fit the training data more closely (possibly at the expense of overtrainingand failing to generalize to the test data). Agent capacity is, in turn, related to the concept of shattering. An agent C(H)can shatter a set of data points Xn, if, for all assignments of labels to those data points there exists a H, so the modelC(H) makes no errors evaluating that set of data points. The capacity of the agent can then be measured by the Vapnik Cher-vonenkis (VC) dimension. The VC dimension of C(H) is the maximum h such that some data point set of cardinality h can beshattered by C(H). The VC dimension, h, can then be used to estimate an upper bound on the test error of C(H) with prob-ability 1 � g, where h is the VC dimension and N the size of the training set:

TestError 6 TrainError þffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffihðlogð2N=hÞ þ 1Þ � logðg=4Þ

N

r: ð2Þ

3.3. Cognitive classes and cognitive diversity

Cognitive diversity results from groups that include intelligent agents from different cognitive classes. Cognitive diversityimplies that the agents think differently. Hong and Page [24] define cognitive diversity with two key dimensions, the agent’sinternal perspective of the problem and the heuristics used by the agent to achieve improved solutions. A brief discussion ofthe nature of cognitive diversity among agents follows. The intention is not to provide rigorous coverage of each cognitiveclass, but to highlight key distinctions in internal perspective and problem solving heuristic. These distinctions are summa-rized in Table 1. We refer to the categories of problem solving methodologies as cognitive classes.

The original Z score model of bankruptcy detection is based on multiple discriminant analysis (MDA) [2]. The MDA cog-nitive class assumes the sample of observations from p1(x), and p2(x) are multivariate normally distributed with a commonvariance–covariance matrix. The nature of the MDA’s solution is to find a hyperplane that effectively separates the healthyand bankrupt classes based on a linear combination of the feature vectors (XA). The heuristic used to obtain a solution is aleast squares calculation of the linear combination XA of feature variables that maximizes the ratio of between-group var-iance to within-group variance.

Like MDA, the cognitive class of logistic regression (LR) assumes normal class distributions with a common covariancematrix. The LR’s internal perspective of the bankruptcy problem is that the posterior probabilities obey a log linear modelof the following form:

logpðk ¼ 2jxÞpðk ¼ 1jxÞ

� �¼ aþ bx: ð3Þ

The posterior log odds of healthy to bankrupt class is a linear function of the feature vector. A solution is obtained by esti-mating the LR parameters a and b by maximum likelihood estimation of a parameter distribution conditioning on X.

The cognitive class Kernel density estimation (KD) perceives the class density functions p1(x), and p2(x) to be locally con-stant. KD defines a Kernel K (i.e., multivariate normal density function) that is a bounded function in feature space with inte-gral one. A nonparametric estimation heuristic is used by KD to obtain an empirical distribution of the class densities p1(x),and p2(x). This empirical distribution of the feature vector within a class gives mass 1/nk to each of the examples in the data.The local estimate of the density p

_

kðxÞ can be found by summing each of these contributions with weight K(x � xi). This canalso be interpreted as an average of Kernel functions centered on each example from the class, the weighted proportion of

Table 1Cognitive classes of intelligent agents.

Cognitive class Perspective Heuristics

Multiple discriminantanalysis

� Partition feature space by hyperplane� Decision boundary (surface)� Based on linear combination of feature vector XA� Parametric: assumption of multivariate normality

� Deterministic for specified training set� XA to minimize ratio of between-group to within-group

variances

Logistic regression � Logit transformation is linear function of X� Estimates posterior class probabilities� Parametric: assumption of normal class distribution and

common covariance matrix

� Maximum likelihood estimation

Kernel densityestimation

� Class density function is locally constant� Kernel is bounded function in X� Nonparametric: estimates posterior class probabilities

� Empirical distribution. . .

� pk(X) is average of Kernel functions. . .

� Dependent on choice of K and sigma

K-nearest neighbor � Kernel is constant over k-nearest neighbors and zeroelsewhere

� Nonparametric: estimates posterior class probabilities

� Posterior distribution is proportion of classes from the near-est k training examples

� Dependent of distance used and choice of k

MLP neural network � Nonlinear projection of X onto a hidden layer (usually ofsmaller dimension)

� Hidden layer neurons interact globally� Nonparametric� Estimates posterior class probabilities� Universal approximation properties

� Back-propagation of error, a gradient descent algorithm� Dependent on initial randomization of weights� Many local minima possible

RBF neural network � Nonlinear expansion of X into an arbitrary (frequentlyGaussian) basis hidden layer

� Linear separation of hidden layer values� Covers theorem� Hidden layer neurons are localized nonlinearities� Nonparametric� Estimates posterior class probabilities� Universal approximation properties

� Clustering locates positions for hidden layer neurons� Output weights determined by linear least squares

optimization� Dependent on parameter choice� Many local minima possible


points around x which have class k. With empirical distributions for the class conditional probabilities, the posterior prob-abilities can be estimated as follows:

p̂ðkjxÞ ¼P

i¼kKðx� xiÞP2i¼1Kðx� xiÞ

: ð4Þ

Agents of the K-nearest neighbor cognitive class (KNN) perceive the problem with a simple adaptive Kernel that is constantover the nearest k examples and zero elsewhere. This suggests a simple estimate of the posterior probability distribution asthe proportions of the classes among the nearest k data points.

The multilayer feedforward neural network (MLP) is a cognitive class designed to mimic (in a crude fashion) biologicalbrain mechanisms. This agent forms an internal representation of the bankruptcy problem by a nonlinear projection ofthe feature variable X onto a hidden layer of neurons, followed by a nonlinear mapping from the hidden layer to two outputnodes that estimate the posterior probabilities of membership in the decision classes. This nonlinear mapping of a singlehidden layer network is of the general form:

pðkjxÞ ¼ fk ak þXj!k

wjkfj aj þXi!j

wijxi

! !; ð5Þ

where j is the number of neurons in the hidden layer, and k in the output layer, wij is the weights between input and hiddenneurons, while wjk is the weights between the hidden and output layers. MLP uses an error back-propagation heuristic toiteratively find improved solutions. During a training process, the heuristic analyzes the squared error gradient with respectto the weights and uses a steepest descent technique to decrease the error.

While MLP constructs a global approximation to a nonlinear input–output mapping, agents of the radial basis function(RBF) cognitive class estimates local approximations to the nonlinear input–output mappings using exponentially decayinglocalized nonlinearities, G. RBF perceives the posterior probability as a linear combination of basis functions. When G isGaussian, RBF can be seen as extending the notion of approximating a probability density by a mixture of known densities:

pðkjxÞ ¼ aþX

j

bjGðkx� xjkÞ: ð6Þ

The norm is unspecified, but is generally a Euclidean or Mahalanobis distance. The heuristic used by the RBF expert to findimproved solutions involves two stages. In the first stage a clustering algorithm (like Self-Organizing Maps, learning vectorquantization, or k means clustering) is used to fix j centers in the data, xj. With fixed centers, the coefficients a and b arecalculated by an ordinary least squares algorithm.


4. Research methodology

Our research methodology is designed to rigorously contrast differences in group decision accuracy between the com-monly used content diversity (manipulation of data) and both ability diversity and cognitive diversity (manipulation ofarchitecture) in group decision processes. In this section, we define the important decisions necessary to conduct this re-search, including the partitioning of the US bankruptcy data set, the formation of bootstrap replicates, agent selection andconfiguration, and controlling the sources of agent ability and cognitive diversity. We then introduce the support vector ma-chine that we employ as a benchmark for assessing the US bankruptcy data accuracy results. To support the generalization ofour findings from the US bankruptcy data we repeat the experiments using a second data set on bankruptcy in Spanish banks[51]. Finally, we describe a methodology to analyze the impact of group composition on decision accuracy.

Before discussing the research methodology, we introduce the formal notation for the two-group bankruptcy classifica-tion problem used in this research. From the population of business organizations, each firm is to be classified as belonging toone of two classes k = {1 = bankrupt, 2 = healthy}. The proportion of bankrupt and healthy firms in the population is p1 and p2,respectively. Knowledge of the recent financial performance of firm i = {1, . . . ,N} is represented by the feature vector Xi. Thefeature vectors from each of the two classes are distributed according to the density function p1(x), and p2(x), the class con-ditional probabilities. The decision task is to classify a firm as belonging to the bankrupt or healthy class on the basis of anobserved value of Xi = x. In the ideal case, both the class densities pk(x) and class prior probabilities pk are known. The famil-iar Bayes classifier for the two-group problem is then an optimal decision model.

pðkjxÞ ¼ pkpkðxÞP2l¼1plplðxÞ

: ð7Þ

This characterizes the posterior probability in terms of the class conditional and prior probabilities. If the decision makers’risk function is to minimize the expected error rate, the optimal Bayes decision rule is to classify a firm into the class with thehighest value of the product of the prior probability times the class conditional probability.

4.1. US bankruptcy data and partitions

The US bankruptcy data is constructed from Standard and Poor’s Compustat database. The feature variables are the set offive key financial ratios originally used by Altman for the Z score model [2]: working capital/total assets, retained earnings/total assets, earnings before interest and taxes/total assets, market value of equity/book value total liability, and sales/totalassets. These ratios are calculated from financial statement information two years prior to a bankruptcy event. There are 329companies in the data set with 93 bankrupt companies and 236 healthy companies. We use labels of �1 (bankrupt) and +1(healthy).

The data is first partitioned into subsets that are used to train the agents (learning data), to prevent overfitting (validationdata), and an independent set of data to estimate generalization error (test set). Our data partitioning methods follow thestrategy used by Breiman [7]. The complete data set of 329 observations is partitioned into a learning set Li with 70% ofthe examples (observations 1–230), a validation set Vi with 15% (observations 231–280), and an independent holdout testset Ti with 15% (observations 281–329). To increase the validity of the results, the data set observations are shuffled andre-partitioned a total of i = 100 times creating 100 different learning, validation, and test sets.

A total of 100 bootstrap replicate training sets LiB are formed from each of the 100 learning partitions (Li) for a total of10,000 bootstrap training replicates. The bootstrap replicates are formed by sampling with replacement from the originaltraining set partition Li. An implication of the bootstrap process is that some of the original training set observations willbe missing in LiB, the bagging training set, while other observations may be replicated several times. Each of the 22 uniqueagents (defined in Section 3.2) is trained on each of the 10,000 bootstrap training replicates and tested on the respective testset. The testing decisions of each agent (total of 490,000 decisions) are stored in a decision database. Specific sampling pro-cedures (defined in Section 4.3) of this database produce groups with different error diversity mechanisms.

4.2. Configuration of intelligent agents

The intelligent agents selected for this research represent most of the agents investigated in prior financial decision re-search as well as those used in commercial applications. These include multiple discriminant analysis (MDA), logistic regres-sion (LR), two different neural network algorithms (multilayer perceptron (MLP) and radial basis functions (RBF)), K-nearestneighbor classifier (KNN), and Kernel density (KD). We use the term ‘‘unique agent” to refer to a decision agent with a uniqueability level, i.e., a specific model with defined parameter values. For example, a k-nearest neighbor classifier with two near-est neighbors is a unique agent. Twenty-two unique agents (defined in the Appendix) are used in this research. With theexception of MDA and LR, all of these agents require specific parameter values; the selection of parameter values is guidedby principles of simplicity and generally accepted practices. For example, the neural network agents are limited to a singlehidden layer, with the number of hidden layer neurons ranging from 2 to 10. A total of five specific abilities (parameter con-figurations) are created for each of these agents (also defined in the Appendix).


4.3. Constructing decision groups with controlled sources of error diversity

We measure the generalization error of the decisions reached by the groups of decision agents at controlled levels of abil-ity diversity and cognitive diversity and contrast the results to baseline conditions from groups that are limited to contentdiversity. All groups formed in this research have a total of 34 members; this quantity exceeds the level of 25 members Brei-man found necessary for effective bagging results [7]. Our strategy is to create a database of agent decision results at eachexperimental condition and to randomly sample the database to create group decisions. We produce a total of 490,000 agentdecision results for each of the 22 unique agents. Three different sampling plans are used to create groups with content errordiversity, content and ability diversity, and content, ability and cognitive diversity. Results are averages of 1000 randomsampling iterations for each of 100 data partitions.

4.3.1. Forming groups with content diversity onlyBy constraining the sampling of database decisions to a specific unique agent, the resulting groups are limited to content

error diversity (the database of decision results consists of unique agent decisions for 10,000 bootstrap learning replicates). Adescription of our specific sampling procedure follows. Take the first agent from the list in the Appendix, MLPa. Start with thefirst data shuffle and randomly identify 34 of its 100 bootstrap replicates by sampling without replacement. Each of the 34replicates serves as one member of the decision group; the result is a matrix consisting of the individual decisions of the 34group members for each of the 49 test examples in the first data shuffle. Majority vote is applied to the 34 member decisionsto produce a group decision for each of the 49 test examples (the reader is referred to Pasi and Yager [43] for a discussion ofmajority opinion in group decisions). The group decision is then compared to the target result to determine a group gener-alization error. The procedure is repeated for shuffles 2, . . . ,100. The process of estimating the generalization error for 100shuffles is then repeated 1000 times yielding an estimate of the generalization error for MLPa. This process is repeatedfor each of the remaining 21 unique agents.

4.3.2. Forming groups with content and ability diversityA second experiment tests the effectiveness of agent ability diversity. This is accomplished by forming groups with

membership constrained to those cognitive classes where multiple architectures or ability levels are possible. There are fourcognitive classes with ability diversity, MLP, RBF, KNN, and KD; each of these classes has five ability levels. For example, toestimate a generalization error for the MLP cognitive class, members of the decision group would include MLPa, MLPb,MLPc, MLPd, MLPe. In this experiment, all 5 of the unique agents in the cognitive class are represented in the group. To filleach group the order of the five unique agents is randomly determined and the agents are replicated a number of times. Thefirst four agents are replicated 7 times and the last agent 6, giving 34 group members. Each time a unique agent is replicatedin a group, a different bootstrap result is randomly selected without replacement from the population of 100 bootstrap rep-licates. Each experiment again yields a decision matrix with 34 members’ individual decisions on 49 test examples. A groupgeneralization error is estimated by using majority vote to determine 49 group decisions and comparing the group decisionsto known targets. Following the same process described in Section 4.3.1 this process is repeated for the 100 data shufflesand for 1000 iterations.

4.3.3. Forming groups with content, ability, and cognitive diversityThis experiment tests the effectiveness of differences in cognitive style by forming groups whose members are randomly

constrained to two and four different cognitive classes, respectively. This experiment uses the process described in Section4.3.1 for data shuffles, error estimation, and number of experimental iterations. The difference in this experiment is the sam-pling procedure. For groups of 2 cognitive classes, the 2 cognitive classes are randomly selected without replacements fromthe set of 4 classes. All 5 unique agents for each of the cognitive classes are included in the group. The groups with 4 cognitiveclasses consist of all 5 unique agents from all 4 cognitive classes. Unique agents are again replicated to create groups of 34members.

4.4. Diversity measurement

Metrics to estimate the amount of error diversity in a group decision process is still an open research issue [10]. None ofthe multitude of diversity metrics proposed is strongly correlated with group decision accuracy. Recognizing these limita-tions, we measure the diversity of each group of agents by the dissimilarity (di(x)) between an agent’s decision (Ci(x)) andthe aggregate decision of the group of agents (C*(x)) as defined by Melville and Mooney [40]:

diðxÞ ¼ 0 if CiðxÞ ¼ C�ðxÞ; otherwise diðxÞ ¼ 1: ð8Þ

To compute the diversity of a group of size B on a training set of size N, an average group diversity measure is calculated

D ¼ 1BN

XB

i¼1

XN

j¼1

diðxÞ: ð9Þ


4.5. Generalization of findings: the support vector machine and Spanish bankruptcy data

We include two other experiments to assess whether the findings of our US bankruptcy data will generalize. First the sup-port vector machine (SVM) is tested on the US bankruptcy. The appeal of SVM is the focus on structural risk minimization fromcomputational learning theory [58]. For this reason, SVM is an ideal candidate to benchmark our US bankruptcy data results.

The second experiment involves repeating the full US bankruptcy data experimental design with an independent data-base of Spanish bank failures [51]. This includes partitioning of the Spanish banks data set, the formation of bootstrap rep-licates, agent selection and configuration, and controlling the sources of ability diversity and cognitive diversity for eachagent. The Spanish bank data are more focused, containing only 66 cases with 9 financial ratios. Bank failures in this dataoccurred during a period from 1977 to 1985.

4.6. Assessing the impact of group composition on decision accuracy

Our final research question investigates the differences in group composition between the most accurate and least accu-rate groups. In this experiment, we repeat the experimental design with different agent sampling constraints; group mem-bership is not constrained by cognitive class but simply by the number of unique agents. Groups are randomly formed with1, 4, 8, and 12 unique agents. This strategy randomizes both ability and cognitive diversity in the group. At any point in thegroup formation process, all agents have an equal probability of joining independent of earlier membership decisions. Wethen retrospectively rank agents by the frequency of occurrence in the most accurate and least accurate groups.

5. Results of agent diversity strategies, US bankruptcy data

Each result reported in this section is based on 1000 random group decision formations for each of the 100 random datapartitions. We first discuss our baseline conditions for this research (the performance of groups limited to content diversity).We then contrast the performance of groups with content and ability diversity, and finally groups with content, ability andcognitive style diversity.

5.1. Research question 1: Accuracy of groups limited to content diversity

We use the term homogenous group for a collection of agents from a single cognitive class with a single ability level. Thereare 22 possible homogenous groups for this research, one for each of the unique agents defined in the Appendix. The onlysource of diversity for these groups is the content diversity generated by the bagging algorithm. This case is a baseline forcomparison with the other forms of diversity investigated. The generalization error (independent holdout test set error) forthe group decisions of each of the 22 homogenous groups is plotted in Fig. 1 for the US bankruptcy data. The average gen-eralization error across all homogenous groups is 0.145. The least accurate homogenous group is the KNNa group with an

y = 0.2617x + 0.1281R2 = 0.5914

0.1

0.11

0.12

0.13

0.14

0.15

0.16

0.17

0.18

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14

X=Melville Diversity

Y=G

ener

aliz

atio

n Er

ror

MLP groups"The experts"

KDe

RBFa

KDd

RBFbRBFe RBFd

RBFc

KDc

KDb

KDa

KNNe KNNdKNNc

KNNb

KNNa

Fig. 1. Generalization error for homogenous group decisions US bankruptcy data.


error of 0.171, while the most accurate homogenous group is the MLPc group with an error of 0.131. All of the MLP homog-enous groups are closely clustered and labeled ‘‘the experts” for this problem since they exhibit the greatest independentability to solve the problem. It is evident from Fig. 1 that there is a significant range of variability in the accuracy of homog-enous groups, with a difference of 0.04 between the most accurate and least accurate group.

One of the striking features of Fig. 1 is that group decision error increases as error diversity (measured by the Melville diver-sity metric) increases. A linear model of the homogenous group decision results (Eq. (11)) has a positive slope estimated to be0.2617 with a p < 0.000. One potential explanation for this phenomenon is research reporting that group accuracy is not alwaysstrongly correlated with error diversity [10]. We also speculate that the homogenous groups exhibit an ineffective trade-offbetween the loss of individual accuracy for the less capable agents and the gain in accuracy from error diversity generatedby content diversity theory. To understand this we must recognize that the capability of the expert in a particular problem re-sults from a match between the problems, computational characteristics and the expert agent’s cognitive style. An expertwhose cognitive style matches the computational requirements for optimal problem solutions exhibits little error diversity.The less capable decision agents exhibit lower capabilities and higher error diversity. The error diversity generated by contentdiversity (in this case bagging) is not adequate to overcome the decrease in individual abilities for the less capable agents.

We found a disproportionate share of the research and applications of decision focused on homogenous groups. Animportant lesson implied by our results for homogenous groups is the critical need to search for the expert decision agent.This imperative will decrease as we examine groups with ability and cognitive diversity in the next sections:

Group Error ¼ 0:2617 �Melville diversityþ 0:1281 ðR2 ¼ 0:5914Þ: ð10Þ

5.2. Research question 2: Accuracy of groups limited to content and ability diversity

Fig. 2 plots the error for groups of decision agents from a single cognitive class but with varying levels of ability as a func-tion of the Melville diversity metric for the US bankruptcy data. These groups, shown as large circles in Fig. 2, consist of deci-sion agents from a single cognitive class (KD, KNN, MLP and RBF cognitive classes) with content diversity and five differentability levels. The error diversity created by varying agent ability results in group decisions that are more accurate than thehomogenous groups that are limited to content diversity only. The average error of all four groups with ability diversity is0.139, compared to 0.145 for the homogenous groups. Our empirical response to research question 2 is that the addition ofability diversity reduces the group decision error by 4.13%. It is also interesting to observe that the KD group of agents withan error of 0.130 achieves more accurate group decisions than the MLP agents (0.133), despite the fact that all five MLPagents were identified as problem experts in Section 5.1.

5.3. Research question 3: Accuracy of groups with content, ability and cognitive diversity

The results discussed in this subsection are groups of decision agents with content diversity, ability diversity, and cogni-tive diversity achieved by sampling from 1, 2, and 4 cognitive classes, respectively. The average generalization errors are

y = -0.7451x + 0.1939R2 = 0.9932

y = 0.137x + 0.1292R2 = 0.25

0.1

0.11

0.12

0.13

0.14

0.15

0.16

X= Melville Diversity Score

Y= G

ener

aliz

atio

n Er

ror

Cognitive diversity: groups sampled from 1, 2, and 4 cognitive classes respectivley

Ability Diversity: groups sampled from unique agents in a single cognitive class

MLP class

KD class

KNN class

RBF class 1 cognitive class

4 cognitive classes

0 0.02 0.04 0.06 0.08 0.1 0.12

Fig. 2. Group generalization error vs. ability and cognitive diverstiy US bankruptcy data.


plotted as squares in Fig. 2. These results offer convincing empirical evidence that the inclusion of decision agents from dif-ferent cognitive classes in the group reduces the generalization error and increases the diversity metric. The group error de-creases from 0.139 for 1 cognitive class to 0.131 for 2 classes and to 0.126 for 4 classes. The slope of the linear model in Eq.(11) for the cognitive diversity groups is estimated to be -0.7451 with a p = 0.053:

Group Error ¼ �0:7451 �Melville diversityþ :1931 ðR2 ¼ 0:9932Þ: ð11Þ

The average error of decision groups formed with 4 cognitive classes (0.126) is the lowest error obtained for all experimentalconditions. Paired two sample mean t tests are used to test for significant differences in means between the 4 cognitive classresults and the following: 2 cognitive class results, KD class with varying levels of ability, and the homogenous MLPc group.There is a statistically significantly difference between the mean generalization error of the 4 cognitive class results and the 2cognitive class result, the KD group and the MLPc homogenous groups, with p < 0.000 in all cases. Our results confirm thatthe diversity of cognitive classes is the most effective method of forming accurate decisions from groups of decision agents.The groups with cognitive diversity are significantly more accurate than groups of homogenous agents and groups from asingle cognitive class with varying ability levels. We also find that groups of 4 cognitive classes are significantly more accu-rate than groups of two cognitive classes.

To summarize our estimates for research questions 2 and 3, the group decision error was reduced by 4.13% by addingability diversity to content diversity. An additional reduction of 5.76% is achieved by adding two cognitive classes to the abil-ity and content diversity sources for a total reduction relative to the baseline content diversity groups of 9.7%. If four cog-nitive classes are added to the ability and content sources, the group error is reduced an additional 9.35% for a totalreduction 13.1%.

5.4. Benchmarking with support vector machine

The generalization error of the SVM on the US bankruptcy data is estimated at 0.154, slightly greater that the averageerror of all homogenous groups (0.144) and significantly higher than the error of the homogenous MLPc group (0.131). Thisdemonstrates that our group decision results outperform the SVM benchmark. It is noteworthy that Huang et al. [27] com-pare SVM and MLP decision agents on US and Taiwan credit markets and they report similar accuracies for both agents.

6. Generalization of findings; Spanish banks bankruptcy data

The results of the Spanish bank analysis are shown in Figs. 3 and 5. The Spanish bank results strongly confirm the earlierconclusions from the US bankruptcy data. Again we observe an increasing error with increasing values of the Melville diver-sity metric for the homogenous groups (triangular markers in Fig. 3). The linear equation of the Spanish bank homogenous

y = -2.0498x + 0.336

R2 = 0.994

y = 1.7279x - 0.0846

R2 = 0.8462

0

0.05

0.1

0.15

0.2

0.25

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16

X= Melville Diversity Score

Y=

Gen

eral

izat

ion

Err

or

Homogeneous groups of a single cognitive class

Cognitive diversity: groups sampled from 1, 2, and 4 cognitive classes respectivley

Ability diversity: groups sampled form unique agents in a single cognitive class

KD class

KNN class

MLP class

RBF class1 cognitive class

4 cognitive classes

Fig. 3. Group generalization error vs. ability and cognitive diversity Spanish bank data.


groups has a slope of 1.7279, with p 6 0.000. The error diversity created by ability (plotted as circles in Fig. 4) produces groupdecisions that are only slightly more accurate than homogenous groups. The average error of all groups with ability diversityis 0.104, compared to 0.107 for the homogenous groups.

Cognitive diversity (sampling of agents from multiple cognitive classes) is the most effective method of generating highperformance groups. As the number of cognitive classes in the group increases, the error decreases from 1 cognitive class(error of 0.104) to 2 cognitive classes (0.073) and finally to 4 cognitive classes (0.057). The linear model for generalizationerror vs. the Melville diversity metric has a slope of �2.0498 with p = 0.049, and R2 = 0.994. This implies that increasing cog-nitive diversity (measured by the Melville diversity score) by a magnitude of 0.01 decreases the group generalization error by0.02. Paired two sample t tests for differences in mean generalization error were analyzed for significant differences betweenthe 4 cognitive class results and the following: 2 class cognitive results, MLP class, and the homogenous MLPe group. All dif-ferences are statistically significant with p values less than 0.000. This again confirms the most accurate groups have mem-bers from all 4 cognitive classes.

Fig. 4 presents the Spanish bank results for groups sampled from constrained levels of unique agents (a proxy for abilitydiversity and cognitive diversity). The collective decision accuracy of the Spanish bank problem is much more pronouncedthan the US bankruptcy problem, with error reductions ranging up to 50%. The average error of all homogenous groups is10.34%, with the most accurate group achieving a 5.30% error and the least accurate a 21.20% error. These errors fall dramat-ically as the number of unique agents in the group decision is increased from 1 to 4. The average error of all randomly formedgroups is 6.66%, or 64.4% of the homogenous average; the most accurate group has a generalization error of 2.90% (54.7% of thehomogenous case), while the least accurate group generalization errors is 18.6% (87.7% of the homogenous case). The accuracyof the average and least accurate groups continues to decrease as the number of unique agents increases to 8 and 12. The aver-age error decreases to 5.51% at 12 unique agents, almost half of the homogenous average. The least accurate group decisiondecreases to 9.4% at 12 unique agents (44.3% of the homogenous case). The error of the most accurate group increases slightlyas agent diversity increases beyond 4 unique agents (2.9–3.3% and 3.4%, respectively for 8 and 12 unique agents).

Clearly the improvements in group decision accuracy are more pronounced for the Spanish bankruptcy data than the USbankruptcy data. The slopes of the group error as a function of the Melville diversity metric is evidence of this difference; theerror improvement for the Spanish data with a slope of �2.05 is significantly greater than the US data at �0.745. While wecannot give a comprehensive reason for these differences, we would like to offer the following insight. First, it is possible thatsome of these differences result from the problem structure. The US data has five feature variables and 329 observations,while the Spanish data has nine variables and 66 observations. If we consider a neural network agent for each problem withsix hidden nodes, the US neural network would have 42 weights trained on 230 observations, while the Spanish agent wouldhave 72 weights trained with 42 observations. The Spanish data is clearly over-parameterized with a parameter: observationratio less that one. This can cause unstable estimates from single agents. However, the tendency to overfit the Spanish mod-els may result in larger group error improvements [14]. We also note that the amount of error diversity in the Spanish data isgreater than the diversity of the US data for all group conditions; the Melville diversity metric for the US data ranges from0.04 to 0.11 and from 0.08 to 0.16 for the Spanish data.

0

0.05

0.1

0.15

0.2

0.25

12841

Number of Unique Agents

Gen

eral

izat

ion

Erro

r

Maximum error-1,000 groups

Average error-1,000 groups

Minimum error-1,000 groups

Homogeneous Groups

Increasing Group Diversity

100%

100%

100%

87.7%

64.4%

54.7%

63.2%

56.7%

62.3%

44.3%

53.3%

64.2%

Fig. 4. Group generalization error Spanish bank data.


7. Results for group composition study

In response to our fourth research question, we report the results of our group composition analysis in this section. FromFig. 5, it is evident that increasing the number of unique agents in a group creates groups with significantly lower decisionerrors for the US bankruptcy data. As the number of unique agents in the group increases to 4, 8, and 12 (Fig. 5), both themaximum and average error decrease monotonically. With 4 unique agents, the maximum error is reduced from 0.171for the homogenous group to 0.153, a reduction of 10.3%; the average error is reduced from 0.1446 for the homogenousgroup to 0.131, a reduction of 9.6%. If we compare the homogenous group with groups of 8 unique agents, the maximumerror is 0.140, an 18.1% reduction; the average error is 0.1289, a 10.9% reduction. The improvement in generalization errorfrom 8 to 12 unique agents is much more modest, but continues to decline for the maximum and average error. The max-imum error for 12 unique agents is 0.1373, an overall reduction from the homogenous groups of 19.5%. The correspondingaverage error is 0.1285, an 11.1% reduction.

We consider the groups with minimum error to be the high performance decision groups; they are the most accurategroups from the set of 1000 randomly formed groups. The minimum generalization error behaves somewhat differently thanthe maximum and average error. At 4 unique agents, the minimum error is 0.1208, a 7.2% reduction from the minimumhomogenous group error. At 8 and 12 unique agents, the minimum error increases slightly to 0.1214 (6.7% reduction) at8 agents, and to 0.1216 (6.6%) at 12 agents. The minimum group error occurs at 4 unique agents and does not decline asthe number of unique agents increases beyond that point. Fig. 3 also confirms that the variability of group results decreasessignificantly as the number of unique agents in the group increases. The range between the maximum and minimum error(0.04) for the homogenous groups decreases monotonically with increasing numbers of agents to 0.016 at 12 unique agents.

To understand the difference in composition between the high performance and low performance groups, we employ asimple weighted accuracy score and rank order each unique agent. The six unique agents that appear most frequently in thehigh performance and low performance homogenous groups for the US bankruptcy data are reported in Table 2. Members ofthe high performance homogenous groups are primarily MLP agents and logistic regression (a closely related algorithm). TheLR and five MLP unique agents are the smartest experts since they individually demonstrate the greatest ability to solve thebankruptcy detection problem. By contrast, members of the low performance homogenous groups are primarily multiple lin-ear discriminant analysis, Kernel density, and k-nearest neighbor agents.

As the group diversity increases to 4 unique agents (increasing both ability and cognitive diversity), the membership ofthe high performance groups changes (Table 2). Although the decision error of these groups has decreased, LR and MLPagents (the smartest experts) now represent only four of the top six positions. Two Kernel density agents (KDa and KDc)are now more frequently associated with high performance groups formed from 4 unique agents. Interestingly, KDa is thesecond least accurate of the 22 homogenous agents (see Fig. 1). Two radial basis function agents (RBFd and RBFb) arenow associated with the low performance groups, in addition to LDA, KD, and KNN agents.

The composition of groups at the two highest levels of cognitive and ability diversity (8 and 12 unique agents) is identicaland strikingly different from the composition of homogenous groups (Table 2). The composition of unique agents in the most

0.1

0.11

0.12

0.13

0.14

0.15

0.16

0.17

0.18

12841Number of Unique Agents

Gen

eral

izat

ion

Erro

r

Maximum error-1,000 groups

Average error-1,000 groups

Minimum error-1,000 groups

100%

100%

100%

89.7%

90.4%

92.8%

81.9%

89.1%

93.3%

80.5%

88.9%

93.4%

Homogeneous Groups

Increasing Group Diversity

Fig. 5. Group generalization error US bankruptcy data.

Table 2Agent membership in high and low performance groups.

High performance groups Low performance groups

Homogenous agent 4 Unique agents 8 and 12 Unique agents Homogenous agent 4 Unique agents 8 and 12 Unique agents

LR KDc KDb KNNa LDA LDAMLPb LR KDa LDA KDe KDeMLPa MLPd KNNa KDa KDd KDdMLPc MLPb KDc KDe KNNd MLPbMLPd KDa KNNb KDb RBFd MLPdMLPe MLPc KNNc KNNb RBFb MLPe

Increasing ability and cognitive diversity.


diverse high performance groups now consists entirely of Kernel density (KDa, KDb, and KDc) and k-nearest neighbor agents(KNNa, KNNb, KNNc). Interestingly, four of these six unique agents are associated with the low performance homogenousgroups, implying they individually have limited ability to solve the problem. It is also noteworthy that three of the MLP un-ique agents (MLPb, MLPd, MLPe), which are the most accurate homogenous groups (i.e., the smartest experts), are now asso-ciated with low performance 8 and 12 agent groups. When we examine the high performance groups we find the lowestgeneralization error is from groups with members drawn uniformly from each of four cognitive classes: MLP, RBF, KNN,and KD. For example, the membership of the single most accurate group formed with 4 unique agents is KDc, RBFe, MLPc,and KNNa; from 8 unique agents the membership is MLPa, MLPc, RBFc, RBFe, KDc, KDb, KNNd, KNNa. This confirms our ear-lier findings; the most accurate groups are sampled from 4 cognitive classes.

Our answer to research question four is that the most accurate cooperative synthesis systems are groups with member-ship compositions that uniformly represent all four cognitive classes. A pattern of expert migration is also noted in theseresults. As the groups become more diverse, the expert agents are replaced in the most accurate groups with agents of lowerabilities. This finding reinforces the concept that ‘‘diversity trumps ability,” a theme advanced by Hong and Page [24], whoconclude that groups of diverse problem solvers can outperform groups of high-ability problem solvers. This finding suggeststhe importance of forming decision groups from a diverse set of agents with differences in ability and cognitive style.

8. Concluding discussion: diversity and group decisions

Our review of the literature suggests that much of the prior research on error diversity in group decision processes fo-cuses on competitive synthesis (tournaments of single agents) and cooperative synthesis with agents limited to contentdiversity (manipulation of data). This research rigorously investigates the value of incorporating ability diversity and cogni-tive diversity into cooperative synthesis systems. We believe this is the first study to accomplish this and are unaware of anyother empirical research on this topic.

Based on our experimental findings, we offer the following insights for practitioners implementing group decision pro-cesses. If the group is comprised of a single decision agent with content (manipulation of data) error diversity, it is criticallyimportant to search for the expert agent, that agent with the highest capability level for the given problem. We find there issignificant variability between and within homogenous groups of a single agent. The inclusion of different ability levels forany single decision agent results in modest improvements in group decision accuracy; we estimate these improvementsrange from 3% to 4%. The most accurate groups had agents from four different cognitive classes. We estimated group decisionaccuracy to increase 11% to 47% from error diversity created by differences in cognitive style. Interestingly, the compositionof the most accurate groups no longer relies on the experts. Therefore, future research should focus on exploring diverse cog-nitive groups; the search for better experts is not productive for cognitively diverse groups.

The experimental results verify that the group error decreases as the number of cognitive classes increases, and that themost accurate groups have agents from four different cognitive classes (i.e., agents that form different perspectives of theproblem and use different heuristics to solve the problem). We show that this strategy reduced the generalization errorby 11% for US bankruptcy data and 47% for Spanish bank failures. With the US bankruptcy data, the groups formed with uni-form membership from 4 cognitive classes achieved a 12.94% reduction in error from the average of 20 homogenous groupsand a 4.65% reduction from the most accurate homogenous group. The Spanish bank results were more striking. The 4 cog-nitive class groups reduced the average error of all homogenous groups by 46.99%, and reduced the error of the most accu-rate homogenous group by 11.29%. The results of this research offer conclusive evidence that multi-agent decision systemsformed from a diverse set of unique agents are significantly more accurate than those formed with a single agent. There is adiscernable decrease in generalization error as the number of unique agents increases.

All sources of group diversity are not universally beneficial. Multi-agent groups that depend solely on content diversityexhibit increasing group decision errors with increasing levels of diversity. In this case, increasing diversity is generated byagents of lower ability that are incorrect more often. By contrast, as the number of cognitive classes represented in multi-agent groups increases, group diversity increases and the group decision error decreases.

It is also interesting to observe the migration of agents as group diversity is increased by increasing the number of uniqueagents in the group. High performance homogenous groups consist of the experts, MLP agents which have the highestindividual ability to solve the problem. The group decision error decreases as the number of unique agents in the group


is allowed to increase. Experts have been replaced in the most effective high performance groups by agents of average ability.Hong and Page refer to this in human groups as ‘‘diversity trumps ability”, groups of diverse problem solvers can outperformgroups of high-ability problem solvers [24].

While we feel the data used in this research is representative of financial applications, the reader is cautioned that ourconclusions are based on one specific application and limited to a two-group binary decision domain with relatively smallnumbers of learning examples. Our results are also dependent on specific decisions including the choice of feature vari-ables, the selection of decision agents and the configuration of agents. While we attempt to use conventional principles inour research, differences in any of these research decisions may produce different results. We focus on understandingsome general properties of diversity and multi-agent decision systems. For this type of research, it is fairly common tomeasure misclassification instances and not misclassification costs. We acknowledge that the costs of misclassificationdo vary considerably between the bankruptcy and survival groups and must be considered during the implementationstage.

Finally, the implications of the no free lunch theorems suggest our results may not be generalized to problem domainswith significantly different problem structures [64]. Essentially, the no free lunch theorem (NFL) establishes that any twoalgorithms are equivalent when their performance is averaged across the set of all possible problems [64,66]. This impliesthat if we know classifier C2 is more accurate than classifier C1 on a specific subset of problems, then C1 must be more accu-rate than C2 on the set of all remaining problems. Recently there has been research defining conditions that obviate the direconsequences of NFL [33,66]. Yu et al. [66] argue that NFL does not eliminate the incentive to continue to improve classifiersin domains where they do well. Koehler [33] establishes conditions for NFL to hold based on the problem structure. In ananalysis of combinatorial problems, Koehler finds that only trivial subclasses of problems fall under the NFL implications.Both of these arguments provide justification for believing that dominant algorithms can exist for specific problem domains.

Appendix. Definitions of unique agents, ability levels, and cognitive classes

Cognitiveclass

Agent andability

Parameter
Note
Neuralnetwork

MLP
MLPa 5 � 2 � 2 MLP = multilayer perceptron, Format I � H1 � O, where I = number of inputnodes, H1 = number of nodes in hidden layer 1, O = number of output nodes
MLPb
5 � 4 � 2 MLPc 5 � 6 � 2 MLPd 5 � 8 � 2 MLPe 5 � 10 � 2
RBF
RBFa 5 � 20 � 2 RBF = radial basis function, Format I � H � O, where I = number of input nodes,H = number of nodes in hidden layer, O = number of output nodes
RBFb
5 � 40 � 2 RBFc 5 � 60 � 2 RBFd 5 � 80 � 2 RBFe 5 � 100 � 2
Parametric
DA MDA MDA = Fisher’s linear discriminant analysis LR LR LR = logistic regression
Nonparametric
KNN KNNa k = 5 KNN = k nearest neighbor
KNNb
k = 7 Format k = i, where i = number of nearest neighbors KNNc k = 9 KNNd k = 11 KNNe k = 13
KD KDa R = 0.1 KD = Kernel density
KDb R = 0.5 Format R = j, where j = radius of Kernel function KDc R = 1.0 KDd R = 1.5 KDe R = 2.0


References

[1] R. Abdel-Aal, Improved classification of medical data using abductive network committees trained on different feature subsets, Computer Methods andPrograms in Biomedicine 80 (2) (2005) 141–153.

[2] E. Altman, Financial rations discriminant analysis and the prediction of corporate bankruptcy, Journal of Finance 23 (1968) 589–609.[3] E. Altman, G. Marco, F. Varetto, Corporate distress diagnosis: comparisons using linear discriminant analysis and neural networks (the Italian

experience), Journal of Banking and Finance 18 (1994) 505–529.[4] E. Bauer, R. Kohavi, An empirical comparison of voting classification algorithms: bagging, boosting, and variants, Machine Learning 36 (1–2) (1999)

105–139.[5] S. Bay, Nearest neighbor classification from multiple feature subset, Intelligent Data Analysis 3 (1999) 191–209.[6] K. Bertels, J. Jacques, L. Neuberg, L. Gatot, Qualitative company performance evaluation: linear discriminant analysis and neural network models,

European Journal of Operational Research 115 (1999) 608–615.[7] L. Breiman, Bagging predictors, Machine Learning 26 (1996) 123–140.[8] L. Breiman, Randomizing outputs to increase prediction accuracy, Machine Learning 40 (3) (2000) 229–242.[9] P. Brockett, W. Cooper, L. Golden, X. Xia, A case study in applying neural networks to predicting insolvency for property casualty insurers, Journal of the

Operational Research Society 48 (1997) 1153–1162.[10] G. Brown, J. Wyatt, R. Harris, X. Yao, Diversity creation methods: a survey and categorization, Information Fusion 6 (1) (2005) 5–20.[11] S. Chen, Computationally intelligent agents in economics and finance, Information Sciences 177 (2007) 1153–1168.[12] S. Cho, Ensemble of structure-adaptive self-organizing maps for high performance classification, Information Sciences 123 (1–2) (2000) 103–114.[13] P. Coats, L. Fant, A neural network approach to forecasting financial distress, Journal of Business Forecasting (1992) 9–12. Winter.[14] P. Cunningham, J. Carney, S. Jacob, Stability problems with artificial neural networks and the ensemble solution, Artificial Intelligence in Medicine 20

(2000) 217–225.[15] S. Davalos, R. Gritta, G. Chow, The application of a neural network approach to predicting bankruptcy risks facing the major US air carriers: 1979–1996,

Journal of Air Transport Management 5 (1999) 81–86.[16] V. Desai, J. Crook, G. Overstreet, A comparison of neural networks and linear scoring models in the credit union environment, European Journal of

Operational Research 95 (1996) 24–37.[17] T. Dietterich, Experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization, Machine

Learning 40 (2) (2000) 139–157.[18] H. Drucker, C. Cortes, L. Jackel, Y. LeCun, V. Vapnik, Boosting and other ensemble methods, Neural Computation 6 (1994) 1289–1301.[19] J. Franke, Bootstrapping neural networks, Neural Computation 12 (2000) 1929–1949.[20] L. Glorfeld, B. Hardgrave, An improved method for developing neural networks: the case of evaluating commercial loan creditworthiness, Computers

and Operations Research 23 (1996) 933–944.[21] L. Hansen, P. Salamon, Neural network ensembles, IEEE Transactions on Pattern Analysis and Machine Intelligence 12 (10) (1990) 993–1001.[22] J. Hansen, Combining predictors: comparison of five meta machine learning methods, Information Sciences 119 (1–2) (1999) 91–105.[23] Y. Hayashi, R. Setiono, Combining neural network predictions for medical diagnosis, Computers in Biology and Medicine 32 (2002) 237–246.[24] L. Hong, S. Page, Groups of diverse problem solvers can outperform groups of high-ability problem solvers, Proceeding of the National Academy of

Science 101 (46) (2004) 16385–16389.[25] J. Hongkyu, H. Ingoo, L. Hoonyoung, Bankruptcy prediction using case-based reasoning, neural networks, and discriminant analysis, Expert Systems

With Applications 13 (1997) 97–108.[26] M. Hu, C. Tsoukalas, Explaining consumer choice through neural networks: the stacked generalization approach, European Journal of Operational

Research 146 (2003) 650–660.[27] Z. Huang, H. Chen, C. Hsu, W. Chen, S. Wu, Credit rating analysis with support vector machines and neural networks: a market comparative study,

Decision Support Systems 37 (2004) 543–558.[28] H. Kim, S. Pang, H. Je, D. Kim, S. Yang Bang, Constructing support vector machine ensemble, Pattern Recognition 36 (12) (2003) 2757–2767.[29] K. Kim, S. Cho, Ensemble classifiers based on correlation analysis for DNA microarray classification, Neurocomputing 70 (1–3) (2006) 187–199.[30] Y. Kim, Toward a successful CRM: variable selection, sampling, and ensemble, Decision Support Systems 41 (2) (2006) 542–553.[31] M. Kearns, Y. Mansour, On the boosting ability of top-down decision tree learning algorithms, Journal of Computer and System Sciences 58 (1) (1999)

109–128.[32] J. Kittler, M. Hatef, R. Duin, J. Matas, On combining classifiers, IEEE Transactions on Pattern Analysis and Machine Intelligence 20 (3) (1998) 226–239.[33] G. Koehler, Conditions that obviate the no-free-lunch theorems for optimization, INFORMS Journal on Computing 19 (2007) 273–280.[34] A. Krogh, J. Vedelsby, Neural network ensembles, cross validation, and active learning, NIPS 7 (1995) 231–238.[35] C. Kun, H. Ingoo, K. Youngsig, Hybrid neural network models for bankruptcy predictions, Decision Support Systems 18 (1996) 63–72.[36] R. Lacher, P. Coats, S. Sharma, L. Fant, A neural network for classifying the financial health of a firm, European Journal of Operational Research 85 (1995)

53–65.[37] K. Lee, I. Han, Y. Kwon, Hybrid neural network models for bankruptcy predictions, Decision Support Systems 18 (1996) 63–72.[38] M. Malhotra, S. Sharma, S. Nair, Decision making using multiple models, European Journal of Operational Research 114 (1) (1999) 1–14.[39] P. Mangiameli, D. West, R. Rampal, Model selection for medical diagnosis decision support systems, Decision Support Systems 36 (2004) 247–259.[40] P. Melville, R. Mooney, Creating diversity in ensembles using artificial data, Information Fusion 6 (2005) 99–111.[41] S. Mukkamala, A. Sung, A. Abraham, Intrusion detection using an ensemble of intelligent paradigms, Journal of Network and Computer Applications 28

(2) (2005) 167–182.[42] D. Opitz, R. Maclin, Popular ensemble methods: an empirical study, Journal of Artificial Intelligence Research 11 (1999) 169–198.[43] G. Pasi, R. Yager, Modeling the concept of majority opinion in group decision making, Information Sciences 176 (4) (2006) 390–414.[44] S. Peddabachigari, A. Abraham, C. Grosan, J. Thomas, Modeling intrusion detection system using hybrid intelligent systems, Journal of Network and

Computer Applications 30 (1) (2007) 114–132.[45] P. Pendharkar, The theory and experiments of designing cooperative intelligent systems, Decision Support Systems 43 (2007) 1014–1030.[46] S. Piramuthu, Financial credit-risk evaluation with neural and neurofuzzy systems, European Journal of Operational Research 112 (1999) 310–321.[47] Y. Raviv, N. Intrator, Bootstrapping with noise: an effective regularization technique, Connection Science 8 (1996) 355–372.[48] A. Refenes, A. Burgess, Y. Bentz, Neural networks in financial engineering: a study in methodology, IEEE Transactions on Neural Networks 8 (1997)

1222–1267.[49] L. Salchenberger, E. Cianr, N. Lash, Neural networks: a new tool for predicting thrift failures, Decision Sciences 23 (1992) 899–916.[50] H. Schwenk, Y. Bengio, Boosting neural networks, Neural Computation 12 (2000) 1869–1887.[51] C. Serrano-Cinca, Feedforward neural networks in the classification of financial information, European Journal of Finance 3 (1997) 183–202.[52] C. Serrano-Cinca, Self organizing neural networks for financial diagnosis, Decision Support Systems 17 (1996) 227–238.[53] R. Sikora, M. Shaw, A multi-agent framework for the coordination and integration of information systems, Management Science 44 (11) (1998) 65–78.[54] M. Skurichina, R. Duin, Bagging for linear classifiers, Pattern Recognition 31 (7) (1998) 909–930.[55] S. Sohn, S. Lee, Data fusion ensemble and clustering to improve the classification accuracy for the severity of road traffic accidents in Korea, Safety

Science 41 (2003) 1–14.[56] F. Tam, M. Kiang, Managerial applications of neural networks: the case of bank failure predictions, Management Science 38 (1992) 926–947.


[57] A. Tsakonas, G. Dounias, M. Doumpos, C. Zopounidis, Bankruptcy prediction with neural logic networks by means of grammar-guided geneticprogramming, Expert Systems with Applications 30 (2006) 449–461.

[58] V. Vapnik, Statistical Learning Theory, John Wiley, New York, 1998.[59] M. Versace, R. Bhatt, O. Hinds, M. Shiffer, Predicting the exchange traded fund DIA with a combination of genetic algorithms and neural networks,

Expert Systems with Applications 27 (3) (2004) 417–425.[60] D. West, S. Dellana, J. Qian, Neural network ensemble strategies for financial decision applications, Computers and Operations Research 32 (2005)

2243–2559.[61] D. West, P. Mangiameli, R. Rampal, V. West, Ensemble strategies for a medical diagnostic decision support system: a breast cancer diagnosis

application, European Journal of Operational Research 162 (2005) 532–551.[62] D. West, Neural network credit scoring models, Computers and Operations Research 27 (2000) 1131–1152.[63] R. Wilson, R. Sharda, Bankruptcy prediction using neural networks, Decision Support Systems 11 (1994) 545–557.[64] D. Wolpert, W. Macready, No free lunch theorems for optimization, IEEE Transactions on Evolutionary Computation 1 (1997) 67–82.[65] E. Yu, S. Cho, Constructing response model using ensemble based on feature subset selection, Expert Systems with Applications 30 (2) (2006) 352–360.[66] L. Yu, S. Wang, K. Lai, An intelligent-agent-based fuzzy group decision making model for financial multicriteria decision support: the case of credit

scoring, European Journal of Operational Research (2007), doi:10.1016/j.ejor.2007.11.025.[67] Z. Yang, M. Platt, H. Platt, Probabilistic neural networks in bankruptcy prediction, Journal of Business Research 44 (1999) 67–74.[68] J. Zhang, Developing robust non-linear models through bootstrap aggregated neural networks, Neurocomputing 25 (1999) 93–113.[69] G. Zhang, M. Hu, B. Patuwo, D. Indro, Artificial neural networks in bankruptcy prediction: general framework and cross-validation analysis, European

Journal of Operational Research 116 (1999) 16–32.[70] G. Zhang, A neural network ensemble method with jittered training data for time series forecasting, Information Sciences 177 (23) (2007) 5329–5346.[71] Y. Zhang, S. Bhattacharyya, Genetic programming in classifying large-scale data: an ensemble method, Information Sciences 163 (1–3) (2004) 85–101.[72] P. Zhilkin, R. Somorjai, Application of several methods of classification fusion to magnetic resonance spectra, Connection Science 8 (1996) 427–442.[73] J. Zhou, D. Lopresti, Improving classifier performance through repeated sampling, Pattern Recognition 30 (1997) 1637–1650.[74] Z. Zhou, Y. Jiang, Y. Yang, S. Chen, Lung cancer cell identification based on artificial neural network ensembles, Artificial Intelligence in Medicine 24

(2002) 25–36.[75] J. Zurada, B. Foster, T. Ward, R. Barker, Neural networks versus logit regression models for predicting financial distress response variables, Journal of

Applied Business Research 15 (1998) 21–30.

http://dx.doi.org/10.1016/j.ejor.2007.11.025

Documents

Diversity of ability and cognitive style for group decision processes