USING NEURAL-NETWORKS TO REDUCE ENTITY STATE UPDATES INDISTRIBUTED INTERACTIVE APPLICATIONS

Aaron McCoy, Tomas Ward, Seamus McLoone and Declan Delaney

Department of Electronic Engineering,National University of Ireland Maynooth,

Maynooth, Co. Kildare, Republic of Ireland.E-mail: { amccoy,tomas.ward,seamus.mcloone } @ eeng.nuim.ie

ABSTRACT

Dead reckoning is the most commonly used predictivecontract mechanism for the reduction of network traffic inDistributed Interactive Applications (DIAs). However,this technique often ignores available contextualinformation that may be influential to the state of anentity, sacrificing remote predictive accuracy in favour oflow computational complexity. In this paper, we present anovel extension of dead reckoning by employing neural-networks to take into account expected future entitybehaviour during the transmission of entity state updates(ESUs) for remote entity modeling in DIAs. Thisproposed method succeeds in reducing network trafficthrough a decrease in the frequency of ESU transmissionrequired to maintain consistency. Validation is achievedthrough simulation in a highly interactive DIA, and resultsindicate significant potential for improved scalabilitywhen compared to the use of the IEEE DIS Standard deadreckoning technique. The new method exhibits relativelylow computational overhead and seamless integration withcurrent dead reckoning schemes.

1. INTRODUCTION

Distributed Interactive Applications (DIAs) are virtualreality systems through which participants can shareinformation via individual and collaborative interactionwith each other and their environment [1]. They offer therealization of simulated virtual worlds that embody amodern extension of communication, encompassing theconcepts of shared time, space and presence [2]. Thedefinition of DIAs includes a wide range of applicationsthat have seen rapid advances in technology and globalpopularity due to the widespread availability and ease-of-use of the Internet [3, 4].

The two key factors that limit large-scale deploymentof DIAs are network latency and bandwidth. Bandwidthrefers to the rate at which data can be communicated perunit time between two end-points, while latency refers to

the delay in communication. High latency and lownetwork bandwidth capacity represent the largestcontributors to the difficulties faced by DIAs insupporting dynamic shared state consistency, potentialscalability and real-time interactivity. A wide variety oftechniques exist that aim to reduce the amount of networktraffic transmitted during the execution of a DIA [2-4].One of the most commonly used techniques is thepredictive contract mechanism known as dead reckoning.

Formally defined within the IEEE Standard forDistributed Interactive Simulation (DIS) [5], deadreckoning reduces network traffic by preventing updatepackets from being sent if the local participant's state hasnot varied from a low-fidelity model of that state by a pre-determined error threshold. It often relies exclusively onthe replication of instantaneous derivative informationbetween controlling hosts for remote entity modeling.However, it ignores available contextual and a prioriinformation that may be influential to the state of anentity, and sacrifices remote predictive accuracy in lieu oflow computational complexity and resource usage [2].

This paper presents a novel extension of deadreckoning, termed neuro-reckoning, that uses neural-networks to allow for expected future entity behaviourwhen transmitting entity state updates (ESUs). Eachcontrolling host employs a set of time-delayed neural-networks trained to predict future changes in entityvelocity over a series of progressively increasingprediction horizons. Future changes in entity location aredetermined using a process of forward simulation throughtime, producing an estimate for the total expected changein entity location over a temporal interval defined by themaximum prediction horizon. On exceeding the errorthreshold, the controlling host issues an ESU containingthe predicted neuro-reckoning velocity vector instead ofthe standard dead reckoning velocity vector. Hence, bydistributing ESUs that implicitly encode future entitybehaviour, neuro-reckoning succeeds in reducing thespatial error associated with remote entity modeling. This,in turn, achieves a reduction in network traffic through adecrease in the frequency of ESU transmission due toerror threshold violation.

1-4244-0657-9/06/$20.00 ©2006 IEEE 295

The remainder of this paper is structured as follows.A mathematical representation of the application domainis given in the next section. This is followed by a detailedexplanation of the novel neuro-reckoning approach inSection 3. A short description of the experimentation isgiven in Section 4, while Section 5 presents the analysisand results, comparing the proposed method with theIEEE DIS Standard dead reckoning. Finally, the paperends with some conclusions and directions for futureresearch in Section 6.

2. THE APPLICATION DOMAIN

To demonstrate and validate our proposed neuro-reckoning framework within an application domainexhibiting a realistic scope, experiments are conductedunder the guise of a simple First-Person Shooter (FPS)style game scenario, designed to produce patterns ofrepeatable human-user behaviour (suitable for trainingtime-delayed neural-networks). The data from this genreof games can be considered a fair representation of thetype of data one would expect to see in commercialnetworked multiplayer computer games [6].

In general, we assume the current state and thecurrent action of an entity at time t to be described by real-valued feature vectors st E 9In and at E 91m respectively.Further, we assume the environmental (i.e. external)influences affecting the state of an entity at time t to bedescribed by a real-valued feature vector et E 9W. Giventhese real-valued feature spaces, reactive modeling of userbehaviour in the discrete-time domain can subsequently beviewed as a problem of function approximation. This isrepresented by the following non-linear input-outputmodel describing the dependence of the future change inentity state i steps ahead on both the current and kprevious states, actions and environmental influences:

2) Environmental State-Space: We assume that anyenvironmental influences are plausibly negligible over asufficiently fine temporal-granularity;

3) Degrees of Freedom Constraints: We assumethat vertical-component space can be ignored byexclusively constraining the manipulation of entity state totwo degrees of freedom along the horizontal plane;

4) Translation and Rotation Invariant: We assumethat changes in entity state are translation and rotationinvariant, meaning that entity reactions are independent ofabsolute location (this ties in with Assumption (2) above).

Assumptions (1) and (2) remove the dependence onthe action and environmental influence state-spaces a ande respectively, reducing Eq. (1) to a problem of multi-variate time-series prediction, represented as:

(t St-1 St-2 .*.**. t-k ) (2)

From our FPS scenario, we assume the state of anentity at time t to be accurately represented by a series oflocation Lt, velocity Vt, and orientation Ot vectors, jointlycomprising a 9-dimensional real-valued state-space:

St = (Lt, Vt, Ot ): Lt, Vt, Ot E=- 9S3 (3)

Assumption (3) removes the dependence on vertical-component space (thus reducing from 9j3 to 9j2), whileAssumption (4) removes the dependence on the locationvector space L, reducing Eq. (3) to a more compact form:

(4)

Ast ([- t],a[t-k, t],e[t-k,t]) 1

Data-based learning problems become exponentiallymore difficult with the increasing size of the state-space[7, 8]. To reduce the complexity of our particular state-space, we employ domain specific knowledge that ispartly realized through a series of consciously introducedapplication constraints, allowing us to make assumptionsregarding the information relevance of state-space featuresfor adequately approximating the functional mappingpresented in Eq. (1). The following assumptions are made:

1) Action State-Space: We assume the consequencesof actions performed by entities are both implicitly andsufficiently encoded within the changes in entity state thatoccur during multiple consecutive time-steps;

By combining Eqs. (2) and (4), we arrive at the finalcompact state-space representation, where k represents thecurrent time-delay and i represents the current predictionhorizon:

(AV~t ,AO°'t )=f(V[t-k,t],o[t-k,t])= f(Vt ... IVt-k Iot I.. IOt-k)

(5)

For the remainder of this paper (and without a loss ofgenerality), we shall restrict our attention exclusively tothe prediction of changes in entity velocity only. Hence, asa final step, we remote the change in entity orientationfrom the output of the dependency relationship in Eq. (5):

AVtt = (Vt ... Vt-k'0t'.t* t-k) (6)

296

St = (Vt,Ot): Vt,Ot C=- 9. 2

3. THE NEURO-RECKONING APPROACH

Eq. (6) can be used to define a set of predictors P for thereactive modeling of user behaviour, represented by:

P = {Pi }, V is {l,...,q }p predicts > Avt+i (7)

Vt+i = Vt + AVt'

In order to realize P, we employ a collection of time-delayed neural-networks (TDNNs) for predicting futurechanges in entity velocity over a series of progressivelyincreasing prediction horizons (up to a total of q stepsahead), as given by Eq. (7) [9]. In this respect, a separateTDNN is trained for each prediction horizon, resulting ina total of q individual networks. Each TDNN consists ofan input layer, a hidden layer, and an output layerarranged in a feed-forward architecture comprising an

additional tap-delay line designed to delay the input signalby a total of k time-steps (see Figure 1), producing a totalof 4(k + 1) input neurons, m hidden neurons, and 2 outputneurons. Non-linear tan-sigmoid transfer functions are

used by neurons contained within the hidden layer, whilelinear transfer functions are used by neurons containedwithin the output layer. Each TDNN utilizes the well-known mean squared error (MSE) performance criterion[7] and is batch trained using the popular Levenberg-Marquardt (LM) update rules for propagating the scaledoutput-error to individual network weights and biases[10]. As per Eq. (4), state vectors reside in a 2-dimensional vector space. As a result, neurons pictured as

residing in either the input or the output layer representtwo components of entity state, physically implemented as

two separate neurons in the final model.

VI

0,

O 3t-k4

Figure 1: Time-delayed neural-network (TDNN) topology for a

single prediction horizon i. Given a maximum prediction horizonq, a total of q such TDNNs (one for each prediction horizon) are

required to predict entity state (see Eq. (7) above). Each TDNNcontains 4(k + 1) neurons in the input layer, m neurons in thehidden layer, and 2 neurons in the output layer.

The set of predictors P provides controlling hostswith the capability to generate a representation for theexpected evolution offuture entity velocity up to a total ofq steps ahead (for each entity under the host's control). Inorder to generate a spatial representation of the expectedevolution of future entity location, controlling hosts can

combine the velocity estimations by a process of forwardsimulation through time (a process we refer to as locally'unfolding' an entity trajectory). Future changes in entitylocation are estimated by simulating the effect of eachpredicted change in entity velocity using a standard first-order, one-step extrapolation equation over a constantsimulation time-step At (equal to the sampling rate of theset of training exemplars used to train the predictors) (seeTable 1). The net effect of this entire process is an

estimate for the total expected change in future entitylocation over the temporal interval z defined by themaximum prediction horizon q and the constantsimulation time-step At, where z = (q + 1)At. Thisprocedure of unfolding is illustrated in Table 1, presentedas an iterative process that is performed over theprogressively increasing prediction horizon i (this process

is performed by a controlling host for a local entity every

time an error threshold violation is detected).

Table 1: The iterative process of unfolding an entity trajectory.

To exploit the process of unfolding, controlling hostsmust compress the resulting predictive state informationfor replication to remote hosts in a compact form that: (1)implicitly encodes the expected evolution of entity state inboth temporal and spatial form, and (2) retains suitabilityfor efficient remote extrapolation of entity state.

Both prerequisites can be satisfied by employing a

suitable 'aggregation policy' designed to summarize thepredictive state information contained within the unfoldedentity trajectory. In theory, an aggregation policy can

assume any functional means intended to reduce theunfolded trajectory to a compact form for replication toremote hosts. On the other hand, the format of theexpected output of an aggregation policy will be pre-defined should maintaining compatibility with existingpredictive contract mechanisms (such as dead reckoning)be considered a priority - in our case, we wish to maintaincompatibility with current standard first-order, one-step

297

Time TDNN Predictor Entity Velocity Entity Location

t V L

t IV> A V{,, = V+ t L,+ LI + VAt...... t. ..... w w. .....

t pi AV,+' V,,t =V + Vt L+1 L+t l + t+ At

.........

i+q P-AVt q V Vt+±AV L -L +V Att + q 1 Lq4q L +VAlt

t+q+l Lr++ =Ll Vt\qAt

extrapolation mechanisms employed within many DIAs.As such, we require an aggregation policy that produces asingle predictive velocity vector, implicitly describing theexpected evolution of entity behaviour over time, whilealso maintaining suitability for replication in the form of astandard entity state update (ESU).

Eq. (8) defines the aggregation policy employedthroughout this paper for use within our proposed neuro-reckoning framework by controlling hosts for eachunfolded entity trajectory. The estimated total expectedchange in entity location (Lt+q+l - Lt) is normalized to aunit vector and scaled by the magnitude of the currentinstantaneous entity velocity at time t (i.e. current speed):

yNR =Lt+q+l -LVtN = ' t lVtl(8)Lt+q+l - (8)

When an entity is deemed to have violated the deadreckoning error threshold, the controlling host issues anESU consisting of current entity location Lt, current entityorientation Ot, and the predictive velocity vector VtNR-Remote hosts subsequently extrapolate entity locationusing a first-order, one-step extrapolation equation definedin Eq. (9). In this way, the reliance of controlling hosts onthe use of neuro-reckoning for replicating predictive stateinformation is entirely opaque to remote hosts, who onlyever view the transmitted ESU packets:

Lt+-= Lt + VtNRAt (9)

4. EXPERIMENTATION

In order to collect the type of data that one would expectto observe in a real-world DIA (and to illustrate thepotential of neuro-reckoning for network trafficreduction), we utilize the commercially available TorqueGame Engine [11] as a customisable research platform forperforming experiments in a controlled manner [6]. Wepresent results for three human-user test subjects (eachpossessing a varying degree of expertise with respect tonetworked multiplayer computer games - Test Subject 1being the most experienced, Test Subject 3 the least), whowere each required to individually compete in a series of12 consecutive experiments (consisting of a pre-specified'hit-point' score limit of 10) against a scripted computer-controlled opponent operating under the influence of asimple dynamic shortest-path behavioural model [12].

Data is collected at a rate of 20 samples per second(i.e. a constant simulation time-step of At = 50ms), andsubsequently normalized in order to improve trainingefficiency and provide good generalization capability. Ofthe 12 experimental datasets collected for each user, thefirst 2 sets were discarded due to the probable appearanceof transient behaviour related to initial learning strategies

adopted by each human-user test subject. In addition, thefinal 2 sets were kept isolated from the data pertaining totraining the neural-networks for the purposes of providingan unbiased means of performance evaluation. Theremaining 8 experimental datasets were combined, afterwhich time a series of exemplars were extracted and thendivided randomly into suitable training (70%), validation(15%), and testing (15%) sets. Based on early evaluationsperformed using various network topologies, the totaltime-delay k was set at 3 (longer time-delays were seen toproduce negligible improvement in training performancerelative to the additional network complexity), and themaximum prediction horizon q was set at 10.

5. RESULTS AND ANALYSIS

Table 2 presents the mean squared error (MSE) trainingperformance over the training, validation, and test sets foreach test subject with respect to the deviation betweenpredicted changes in entity velocity and the correctchanges in entity velocity over each set of TDNNpredictors. Each individual TDNN learns to predict over aspecific prediction horizon, where the training procedurecontinues until the validation set determines the 'early-stopping' point for maximum generalization [7] (oralternatively, the training period reaches 100 epochs). Ascan be observed, each of the datasets exhibits similarresults over all of the prediction horizons, indicating goodgeneralization ability across the entire set of predictors.Evident from inspection of Table 2 is the fact that as theprediction horizon increases, we can observe anapproximately linear trend in the decreasing accuracy ofeach neural-network. Hence, the predictive performanceof each neural-network is directly proportional to theprediction horizon.

TsSubjec1Prediction e ampE (9618 S>amples)Training Validation test00 04 0.0005 00004

2 0001 0.005 00 153 0 0.0033 000314 00055 0009 0054

00M79 0~00 076 010 0.0111 ,.0100

011 00133 0 18 i001 0.0154 001 369 0.0160 0.0174 0050101.074 0.0193 0.i66

Test Subect 2(10114 Samples)

Trainig EVaidatio Test

030 >9 t0.0041 0 0401.00E9 E0.002 0.0700099 00O04 O010410131 E00135 040138

.0029 E0.029 00.249

.0 248 0 0261 0.0270

Test Subject 3(9835 Samples)

Trining Vyalidatio e0000 00 3 0 X00040.0070 0,0010 0.001 7M0022 O2OO24 0.00260.0037 O039 0004400057 D00 9 000iO630080 O0087 0.001

00135 0.0140 0.01450.0157 00.166 E001720017 00198 0.0202

Table 2: MSE results over training, validation and test sets foreach individual human-user test subject.

Table 3 presents simulation results that wereconducted over a series of increasing error thresholds forTest Subject 1, Test Subject 2, and Test Subject 3. Allpresented error threshold values are measured in terms ofTorque world units. To put the error thresholds intoperspective, the height of an entity within our testenvironment is approximately 2.3 Torque world units.Ideal network conditions were assumed to ensure unbiasedevaluation of the performance of the first-order, one-step

298

Threshold

0.51

1.522.533.544.551025

TetSbet1(19Iamls etSbet2(279Smls etSbjc 347Smls

DR26941823142312171064974872786733679422221

NR2392165612971111994893809742699650400218

% Red-11.2-9.2-8.9-8.7-6.6-8.3-7.2-5.6-4.6-4.3-5.2-1.4

DR4103277822111869163914811352123211551092683325

NR3717243619561668147013181234113210771013661312

% Red-9.4-12.3-11.5-10.8-10.3-11.0-8.7-8.1-6.8-7.2-3.2-4.0

DR4309300523972025179116181481136412841186753368

NR3926242619611665151313651267117211041051694350

% Red-8.9-19.3

=-18.2=-17.8

-15.5-15.6-14.4-14.1-14.0-11.4-7.8-4.9

Table 3: Entity state updates generated and totalled over all 12 recorded trajectories for each individual test subject (see Section 4).

dead reckoning ( DR ) model and the proposed first-order,one-step neuro-reckoning ( NR ) technique.

Throughout Table 3, packet counts for both scenariosare listed under the headings DR and NR respectively.The term % Red refers to the percentage reduction (orincrease) in the number of ESU packets generated due toerror threshold violation, where negative values areindicative of superior performance by the proposed neuro-reckoning framework in comparison with the standarddead reckoning model. Packet counts are generated andtotalled over all of the 12 recorded trajectories for eachindividual test subject (see Section 4), and all simulationswere performed at a constant rate of 20Hz (i.e. the originalsampling rate of the data) using a maximum predictionhorizon of q = 10 (i.e. 10 TDNNs).

From inspection of the results, it is noted that in everysituation, neuro-reckoning offers a reduction in thenumber of ESU packets generated that ranges from smallbandwidth savings (in the region of 2% or lower packetreduction) to very large bandwidth savings (in the regionof just under 20% packet reduction). Furthermore,percentage reductions in the number of ESU packetsgenerated are observed over the entire series of simulatederror thresholds. Of particular interest are the large packetreductions typically observed at lower error thresholds,implying excellent potential for use within DIAs requiringa high degree of tightly coupled synchronization betweenlocally and remotely modeled entity state. In general,there appears to be an approximately linear trend betweenthe (decreasing) reported percentage reductions and the(increasing) error threshold. This implies a relativelystable predictive accuracy with respect to the grossperformance gain (i.e. expected overall packet reduction)resulting from the use of our proposed neuro-reckoningframework in contrast to the DIS dead reckoning model. Itshould be noted that although the number of users limitsour ability to generalize the results, it is sufficient fordemonstrating the efficacy of the neuro-reckoningtechnique for cases where good neural-network predictionis available for a user.

Figure 2 (a)-(b) presents the results generated usingboth ESU mechanisms for Test Subject I over an errorthreshold of 0.5 Torque World Units (TWUs), visuallyillustrating the potential of neuro-reckoning for furthernetwork traffic reduction over a small sample section fromone of the trajectories recorded for that user.

io,da .4i..r] (dash.d) -ests -enote dead -ek-ixg (,,.]id) 4*t.,ai-ttr

-20,

40 'LFi

-60~~~~~~~~~~~~~~~~~~~~~~~~X,,-

40~~~~~~~~~~~~~~~~~~,..

~~~~~~~80~~ ~ ~ ~ ~ ~~,t

-30 _-

-40

-60

86 9 2 9 9 98 1 ltoo 102 10 l-o- M

X~~~~~~~~~ OMb

ebFis-rde er-eknig(R 1 SU eeae)Fiur 2 a-b:EUpce eeainrslsfrTs ujcIve an0ero hehl f0TrqeWrdUis(Ws oa Pmlaml seto xratdfo rcre sr rjcoy

299

Test Subject 1 (21595 Samples) Test Subject 2 (27749 Samples) Test Subject 3 (32477 Samples)

In both cases, the remotely modelled entity trajectoryis represented as a solid line that is overlaid on top of theoriginal trajectory represented as a dashed line, where thenumber of solid black stars represents the number of ESUsgenerated due to error threshold violation.

From inspection of the spatial plots, we can observehow first-order neuro-reckoning accurately compensatesfor expected changes in future entity behaviour, reducingthe number of ESUs required to maintain an equivalentspatial consistency to that provided by the first-order DISdead reckoning mechanism from 20 to 16.

6. CONCLUSIONS AND FUTURE WORK

The proposed neuro-reckoning model provides a statisticalfoundation for the modelling and prediction of repeatablepatterns of human-user behaviour [13, 14]. By forecastingexpected future entity behaviour in advance, anddistributing this information to remote hosts in a compactform, neuro-reckoning succeeds in reducing the spatialerror associated with remote modelling of entity statewhen compared with the traditional use of first-orderinstantaneous derivative information. Consequently,neuro-reckoning achieves a reduction in the number ofESUs generated due to error threshold violation.

Presented simulation results validate the potential ofour proposed framework for improving the accuracy ofremote entity modelling over the use of a pure deadreckoning approach throughout our experimental set-up,exhibiting a reduction in network bandwidth requirementsover a wide range of error thresholds and indicating viablepotential for satisfying many spectrums of consistencyrequirements throughout a broad range of applications.

Future work includes the investigation of confidence-interval estimation methods to quantify the reliability ofthe predictions made by each set of neural-networks (witha view to dynamic model switching) [15], in addition to acomplexity analysis for the use of the neuro-reckoningalgorithm by controlling hosts during real-time execution.Further validation of the technique on more test subjects,comprising a wider array of scenarios involving multiplehuman-users, complex real-world environments, andrealistic network conditions, will also be undertaken.

7. ACKNOWLEDGEMENTS

This material is based upon work supported by theScience Foundation of Ireland and Enterprise Irelandunder grant no. IRCSET/SC/04/CS0289, and also byEnterprise Ireland under grant no. SC/2002/129.

8. REFERENCES

[1] E. F. Churchill, D. N. Snowdon and A. J. Munro,Collaborative Virtual Environments: Digital Places and Spacesfor Interaction. London, UK: Springer-Verlag, 2001.

[2] S. K. Singhal and M. J. Zyda, Networked VirtualEnvironments: Design and Implementation. New York, NewYork: Addison-Wesley, ACM Press SIGGRAPH Series, 1999.

[3] D. Delaney, T. Ward and S. McLoone, "On Consistency andNetwork Latency in Distributed Interactive Applications: ASurvey- Part I," Presence: Teleoperators and VirtualEnvironments, vol. 15, no. 2, pp. 218-234, April 2006.

[4] A. McCoy, D. Delaney and T. Ward, "Game-State FidelityAcross Distributed Interactive Games," ACM Crossroads, vol.9.4 (Networking Issue), pp. 4-9, Summer 2003.

[5] IEEE Standardfor Distributed Interactive Simulation -Application Protocols, IEEE Standard 1278.1-1995, 1995.

[6] A. McCoy, D. Delaney, S. McLoone and T. Ward,"Investigating Behavioural State Data-Partitioning for User-Modelling in Distributed Interactive Applications," in Proc. 8thIEEE International Symposium on Distributed Simulation andReal-Time Applications (DS-RT '04), Budapest, Hungary,October 2004, pp. 74-82.

[7] J. C. Principe, N. R. Euliano and W. C. Lefebvre, Neural andAdaptive Systems: Fundamentals through Simulation. NewYork, NY, USA: John Wiley and Sons, Inc., 2000.

[8] G. I. Webb, M. J. Pazzani and D. Billsus, "Machine Learningfor User Modeling," User Modeling and User-AdaptedInteraction, vol. 11, no. 1-2, pp. 19-29, March 2001.

[9] R. Bone and M. Crucianu, "Multi-step-ahead Prediction withNeural Networks: A Review," Approches Connexionnistes enSciences Economiques et en Gestion, vol. 2, pp. 97-106,November 2002.

[10] M. T. Hagan and M. B. Menhaj, "Training FeedforwardNetworks with the Marquardt Algorithm," IEEE Transactions onNeural Networks, vol. 5, no. 6, pp. 989-993, November 1994.

[11] J. Lloyd, "The Torque Game Engine", Game DeveloperMagazine, vol. 11, no. 8, pp. 8-9, September 2004.

[12] A. McCoy, D. Delaney, S. McLoone and T. Ward,"Dynamic Hybrid Strategy Models for Networked MultiplayerGames," in Proc. 19th European Conference on Modelling andSimulation (ECMS 2005), Riga, Latvia, June 2005, pp. 727-732.

[13] W. S. Sarle, "Neural Networks and Statistical Models," inProc. 19th Annual SAS Users Group International Conference,SAS Institute Inc., Cary, NC, USA, April 1994, pp. 1538-1550.

[14] I. Zukerman and D. W. Albrecht, "Predictive StatisticalModels for User Modeling," User Modeling and User-AdaptedInteraction, vol. 11, no. 1-2, pp. 5-18, March 2001.

[15] G. Papadopoulos, P. J. Edwards and A. F. Murray,"Confidence Estimation Methods for Neural Networks: APractical Comparison," IEEE Transactions on Neural Networks,vol. 12, no. 6, pp. 1278-1287, November 2001.

300