Robustness Analysis of Artificial Neural Networks and Support Vector Machine in Making Prediction

Saiful Anwar Department of Accounting

STIE Ahmad Dahlan, School of Finance and Banking Jakarta, Indonesia

olieanwar@gmail.com

Rifki Ismal Department of Economics

Faculty of Economic, University of Indonesia Jakarta, Indonesia

rifki_ismal@yahoo.com

Abstract—This This study aims to investigate the robustness of prediction model by comparing artificial neural networks (ANNs), and support vector machine (SVMs) model. The study employs ten years monthly data of six types of macroeconomic variables as independent variables and the average rate of return of one-month time deposit of Indonesian Islamic banks (RR) as dependent variable. Finally, the performance is evaluated through graph analysis, statistical parameters and accuracy rate measurement. This research found that ANNs outperforms SVMs empirically resulted from the training process and overall data prediction. This is indicating that ANNs model is better in the context of capturing all data pattern and explaining the volatility of RR.

Keywords;Artificial Neural Networks, support Vector Machine, Rate of Return, Islamic Bank.

Introduction

Recently, the authors in forecasting are have been encouraged to apply machine learning technique such; as neural networks, genetic programming and support vector regression, particularly, when traditional statistic techniques perform low performance to deal with nonlinearities and non stationary of economic and financial data. Lately, neural networks model has attracted many researchers to be applied as an alternative and becoming more popular. This is because the model outperforms traditional statistic technique in terms of explaining variance ability and predictive accuracy by minimizing training error. On the other hand, the utilization of SVMs model in making prediction is also interesting to be investigated since the model proposes a different point of view. In spite of minimizing training error, the SVMs model uses structural risk minimization principle to reduce the expected error of learning machine and problem of over fitting [9].

This research aims to examine the robustness of those models in learning and making prediction of depositor return of Indonesian Islamic bank. We justify the importance of this research according to the following reasons: First, the time deposit product of Indonesian Islamic bank is developed using the profit and loss sharing principle which return delivered is uncertain according to the bank’s profitability. Second, prior studies suggested that Indonesian people in patronizing Islamic banks is expecting better return rather than religious motive. Third, they freely put or withdraw their monies since the banks charge no

penalty when they want to break their time deposit account at any time. Therefore, finding the most reliable model to predict one or two months ahead of time deposit return is very important to facilitate the depositor’s profit motive.

I. LITERATURE REVIEW

A. The role of macroeconomic variables This study adopts the bank failure theory which issued by

[6]. The theory said, “Extremely bad management may not prove fatal to a particular bank until economic condition leads to unexpected capital outflows or loan losses”. It means that the performance of the bank is mainly influenced by economic condition. Therefore, this research uses only macroeconomic variables to predict Islamic bank’s performance which is reflected by time deposit return delivered to depositors. Generally, the macroeconomic variables determine bank’s profitability in the following ways: [5] reports that market expansion would enable banks to increase profits as represented by a strong relationship between money supply and profit. Furthermore, stock indices may lead to a higher growth of the firm, industry and country level. Such conditions will give more profit to the banks from financing activities. Meanwhile, [3] informs that inflation positively affects the bank’s profits if the revenue that accrues from business is larger than the arising of overhead cost due to inflation. Interest, on the other hand, affects the majority of funding and financing activities of a bank and later on the bank’s profit. Lastly, exchange rate does not affect profits of Islamic bank from foreign exchange trading, as it does to conventional bank, since it is prohibited. This benefits Islamic bank through its impact on the fluctuation of the price of goods that affect business trading and market.

B. The uniqueness of Islamic bank time deposit One of the main sources of fund in Indonesian Islamic

banks is called mudharabah time deposit. The banks use the fund to finance the business which is not prohibited by Islamic law and lately share the profit with depositors [8]. Unlike interest that provides fix regular revenue, the return of mudharabah time deposit which is represented by monthly rate of return is uncertain. Additionally, the rate is varied among Islamic banks since it depends on the bank’s profitability and pre-agreed profit and loss sharing (PLS) ratio offered when depositors open the account. Every

Ninth IEEE International Symposium on Parallel and Distributed Processing with Applications

978-0-7695-4428-1/11 $26.00 © 2011 IEEE

DOI 10.1109/ISPA.2011.64

256

Ninth IEEE International Symposium on Parallel and Distributed Processing with Applications

978-0-7695-4428-1/11 $26.00 © 2011 IEEE

DOI 10.1109/ISPA.2011.64

256

month, the Islamic banks publish the RR report to assist depositors, comparing the current rate of return with current market interest rate.

C. The Theory of artificial neurral networks model Accumulated studies have shown that ANNs is better

than traditional statistic model in making prediction. [13] reported that ANNs outperforms MLR in predicting housing value. [1] reported the superiority of ANNs against multi discriminant analysis model (MDA) in predicting bankruptcy of 128 firms. In detail, ANNs achieves correct classification accuracy in the range of 77.8% to 81.5%, while MDA’s accuracy is in the range of 59.3% to 70.4%. Additionally, [2] review 89 studies on corporate bankruptcy prediction. They found that ANNs achieve averagely 88% accuracy rate while traditional statistical models achieve 84% accuracy rate. Moreover, [12] confirmed the previous findings that ANNs is more powerful in predicting bank performance compared with multiple linear regressions using the mean square error (MSE) and mean square prediction error. Although ANNs model has demonstrated some successes in this area, there has been little work done on forecasting in Islamic banking and finance research using ANNs model.

Actually, ANNs is a part of machine learning technique which tries to simulate the way of learning of the human brain in making decisions. Its function mimics biological neurons in which the structure consists of a group of artificial neurons which are interconnected, creating networks. The main elements of neuron for learning or information processing are: Inputs, Weight, Summation Function, Transformation function and output. Similar to the human brain, the process of training the networks is needed to recognize patterns, to develop generalization and to learn to improve the performance. Accordingly, ANNs is powerful to solve problem in areas of prediction and classification.

Technically, the process of ANNs is briefly explained as follows. Initially, there is a neuron j which has a certain number of inputs (x1,x2, x3,…xj) and single output (yj). Each input has a weight (w1j, w2j, w3j,...wij) as an indicator of importance of the incoming signal into neuron (j). The net value (uj) of the neuron is then calculated with the sum of all the input value multiplied by their respective weight. Further, with reference to a threshold value (tj) and activation function, the neuron (j) determines an output value (yj) which will be sent as output to other neuron. Each neuron has its own unique threshold value (tj). If the net value (uj) is greater than the threshold (tj), the neuron (j) will send output (yj) to other neurons. In addition, the activation function is a function used to transform the activation level of a unit (neuron) to an output signal. Currently, sigmoid and logistic are the most popular activation functions used.

The typology of most neural networks generally involves a collection of neurons that are configured in two or more layers. Therefore, the research combines some neurons into multilayer structures to have the power of pattern recognition and prediction. Accordingly, this research employs a multi layer feed-forward networks which is the

most common type of neural networks currently in use. The multi layer feed-forward network comprises of input layer, hidden layer and output layer. Specifically, the input layer is a layer that is directly connected to outside information. All data in the input layer will be feed-forwarded to the hidden layer as the next layer. Meanwhile, the hidden layer functions as feature detectors of input signals and releases them to the output layer. Finally, the output layer is considered as a collector of the features detected and as a producer of the response. In the networks, the output from output layer is the function of the linear combination of hidden unit’s activation; in which the hidden unit’s activation function is in form of a non-linear function of the weighted sum of inputs. Mathematically, the ANNs model can be written as in the following:

y = f(x,θ) + ε (1) Where x is the vector of explanatory variables, θ is

weights vector (parameters) and ε is the random error component. Then, eq. 2, the unknown function, is used for estimation and prediction from the available data. As such, the model can be formulated as:

⎥⎥⎦

⎤

⎢⎢⎣

⎡∑ ⎟⎟

⎠

⎞⎜⎜⎝

⎛∑++=

= =

m

jj

n

iijij vwxhvfY

1 10 λ (2)

Where: Y = network output f = output layer activation function v0= output bias m= number of hidden units h= hidden layer activation function λj= hidden unit biases (j = 1,. . . ,m) n= number of input units xi= inputs vector (i = 1,. . . ,n) wij= weight from input unit i to hidden unit j vj= weights from hidden unit j to output (j = 1,. . . ,m)

D. The theory of support vector machines Support vector machine is a model proposed by Vladimir

Vapnik and Carolina Cortes in 1995. In spite of minimizing training error as ANNs model does, this model is intended to minimize an upper bound of the generalization error to achieve generalized performance by utilizing the structural risk minimization principle. The upper bound of generalization error is the combination of the training error and a regularization term which controls the complexity of the hypothesis space. Initially, the model which implements a learning algorithm is intended to recognize pattern by mapping the nonlinear of a complex data set into higher dimensional feature space (see figure 1). However, with the introduction of Vapnik’s ε -insensitive loss function, the model has been extended to be used in making prediction which is also called as support vector regression [15]. As a newly developed algorithm, many researchers found satisfying results in making prediction especially in financial and economic field. For example, [10] employed SVMs to predict exchange rate of US dollar against Japanese Yen.

257257

Figure 1. Function of Φ (x) maps input into higher dimensional feature space

(Free source: Kuliah Umum InfoKomputer.com)

Additionally, [7] used SVR combined with genetic algorithm to predict housing price in China. [15] explains the derivation of model mathematically to make prediction of Yi from a given data set; G = {(xi, ai)} N

i 1= where xi is the vector of model input, ai is actual value of input and N is the total number of data pattern, as follow:

bxwxfY iii +== )()( φ (3) Where: Yi = the corresponding scalar output.

)(xiφ = the feature space of inputs x wi and b = estimated coefficients using structural risk

minimization principle. While, the structural risk minimization principle used is

denoted as: 2

1 21),(1)( wYdL

NCCR ii

N

i+∑=

=ε (4)

Where: |di-Yi|-ε, |di-Yi| ≥ ε,

=),( ii YdLε 0, others, C = regularization constant ε = precision parameters representing the distance of the tube located around regression function. d = the actual value at period i Y = the estimation value at period i

=),( ii YdLε Vapnik’s ε-insensitive loss function which will equal zero if Yi is within the ε-tube.

Furthermore, 2

21 w which so-called the norm of weight

vector is used to calculate the flatness of the function. This is the regularization term that used to control the trade-off between the complexity and the approximation accuracy of the regression model [7]. Meanwhile, C in eq. 5 is used to specify the trade-off between risk and model flatness. Next, the variables ζ and ζ* will be used to represent the distance from point of actual values to the corresponding boundary values of ε-tube as depicted in figure 2. Then, the eq. 5 is transformed into the constrained form as follows:

Minimize: R(w, ζ, ζ*)= ∑+=

N

iNCw

1

2 *) +(121 ζζ (5)

Subject to: ,)( *

iiiii dbxw ζεφ +≤−+ i=1,2,..N ,)( iiiiii bxwd ζεφ +≤−− i=1,2,..N

ζi, ζi* ≥ 0, i=1,2,..N

Figure 2. Parameters of Support Vector Machine

The constrained optimization problem is solved as denoted in eq.7 using Langrangian form (αi, αi*) which satisfy equality αi x αi*= 0 where αi≥0 and αi*≥0

L(wi, ζ, ζ*,αi, αi*,βi, βi*) = ⎟⎟⎠

⎞⎜⎜⎝

⎛∑=

+N

iNCw

1*) +(12

21 ζζ

- [ ]∑ ++−+=

N

iiiii dbxw

1i )( ζεφα

- [ ]∑ ++−−=

N

iiiiii bxwd

1

** )( ζεφα - ∑=

N

iiii

1

**i ) +( ζβζβ (6)

Furthermore, in order to make regression, the Karush-Kuhn-Tucker (K) conditions are applied in which resulting the dual Langrangian as follows:

J(*, ii αα ) =

),(21

1 111i

N

i

N

j

*jj

*ii

*i

N

iii

*i

N

iii xxK)α(α)α(α)α(αdε)α(αd ∑∑ −−−−−∑−−∑

= ===

Subject to:

,0)( *

1=−∑

=i

N

ii αα ,),(0 * Cii ≤≤ αα i=1,2.3…N (7)

In this equation, the Langrangian multipliers which are represented by iα and *

iα satisfy the condition where iα x *iα = 0. Then, the optimum expected weight vector of the

regression hyper plane (w*) is determined by:

),()(* *

1jii

N

ii xxKw αα −=∑

=

(8)

Finally, the SVMs model used for regression is formulated as follow:

bxxKxf jiiN

ii +−∑=

=),()(),,( *

1

* αααα (9)

The Kernel function in eq. 10 is represented by )()(),( jiji xxxxK φφ ×= which values are equal with the

inner product of vector xi and xj in the feature space )( ixφ and )( jxφ . Additionally, according to Karush-Kuhn-

Tucker conditions, only some of )( *ii αα − will be equal

with 0. This research employs radial basis function (RBF) for Kernel type which is the most widely used because of its localized and finite responses across the entire range of the real x-axis (Statistica user guide). The function is denoted as follow:

258258

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛ −−= 22

exp),(σ

jiji

xxxxK (10)

Where; σ is the width of the RBF and also determined by user like C and ε.

II. DATA AND METHODOLOGY

A sample data set consisting six type of macroeconomic variables for the period of January 2000 – April 2010 are used as independent variables such as exchange rate of US dollar against Indonesian rupiah (EXCH), stock indices (STIN), money supply (represented by M1), inflation rate (INFR), market interest rate of one month time deposit (INTR), and central bank’s interest rate (BIRT). Meanwhile, the data of average rate of return of one-month mudharabah time deposit of all Indonesian Islamic banks (RR) from the same period are used as dependent variable. All data are collected from Indonesian central bank (Bank of Indonesia). In the beginning, those data are used to build and train the prediction model developed by ANNs and SVMs model. Furthermore, the performance of each model is evaluated using following ways: 1. Analyzing actual vs. predicted graph resulted from the training process. 2. Analyzing robustness of the model built using statistical parameters. 3. Calculating accuracy rate resulted from out of sample data prediction covering period of January 2009 until April 2010.

A. Working with artificial neural networks In the beginning, all data are preprocessed to simply

convert the input data into a new version for three reasons [4]. (1) To ensure the size of data reflects the importance level in determining the output. (2) To facilitate the random initialization of weights before training the networks. (3) To normalized all data to avoid different measurement due to different unit of input. Next, Alyuda software provides an exhaustive search feature to design the neural networks architecture. As a result, this research use N(8-6-1) for learning and testing process which will be conducted later on. The structure of networks selected consists of 8 neurons in input layer, 6 neurons in hidden layer and 1 neuron in output layer. Additionally, the configurations used for learning process are as following: (1) the logistic function is selected for all neuron. (2) The sum-of-squared errors are selected to minimize the output errors. These are summation of squared differences between the actual values and model’s output. (3) The networks outputs are set up between 0 and 1 due to logistic activation function used. In the next step, the networks are trained with specific condition to avoid over fitting such as using back propagation as learning algorithm, the learning and momentum rates are set at 0.1, and for completeness, the process should be stop when mean squared error reduces by less than 0.000001 or the model completes 20,000 iterations, whichever condition occur first.

The quality of networks is investigated by using statistical criteria as shown in table 1. The value of correlation (r) and R2 are the indicators of multiple correlations between independent variables and dependent variable. The coefficients can range from -100 to +100 percent. When the coefficients are closer to 100 percent indicates the positive linear relationship between both variables become stronger. Meanwhile, mean of ARE (in absolute value) is error value that shows the quality of training process. It means that the smaller the error is the better quality of the networks will be. Finally, the prediction process will be conducted using the selected networks.

TABLE 1. THE QUALITY OF NETWORKS

B. Working with support vector machine In the beginning, the data is partitioned into two parts;

108 data covering period January 2000 until December 2008 are used for training, and the other 16 data covering January 2009 until April 2010 are used for testing. This research employs regression type 1 to initiate training as provided by Statistica software. The training process consists of two stages. The first stage is to estimate value of C and ε that achieves the lowest regression error. And, the second stage is to train the model using all training sample based on value C and ε found in the prior stage. Fortunately, the software provides v-fold cross validation check box to determine the best value of the training parameters specified in the Grid search group box. Actually, the Grid search is a straightforward method using exponentially growing sequences of Capacity (C), and Epsilon (ε) to identify good parameters in which it will select the best parameter of C and ε with the lowest regression error [11]. In this experiment, the training process is limited according to the following conditions which ever condition occurs first: 1. Maximum number of iteration (1000 iterations). 2. Value of training error achieved (0.001). 3. Size of cache (40 Mb). Beforehand, the RBF is already selected as Kernel type which value of σ is 0.143. As a result, the second stage determines value of C, ε, number of support vectors and cross-validation error which are 10, 0.1, 72, and 0.059 respectively. Meanwhile the quality of model is shown by value of mean error squared (0.697), standard deviation ratio (0.538) and R2 (0.848) which indicates that the chosen model is robust with respect to the selected model variables and coefficients. Finally, according to this result, the out of sample prediction is performed.

C. Performance measurement The research uses three methods to compare the

performance of each model as follow: (1) Comparing graph

Data set Correlation (r) in %

R2 in %

Mean of ARE

Training data set 93.89 86.94 0.067 Validation data set 92.94 79.64 0.087 Testing data set 90.32 78.29 0.079 All data set 0.06 84.51 0.071

259259

of actual and predicted RR resulted from the training process. (2) Comparing three statistical parameters such as R2, mean absolute error (MAE) and normalized means squared error (NMSE). Actually, NMSE and MAE are parameters used to evaluate how close the prediction results to the eventual outcomes. (3) Investigate accurateness of models in making prediction using out of sample data according to eq. 11, as follows:

Accuracy power = 100% - % Error of prediction (11)

III. RESULTS As shown in fig. 3, it suggests that ANNs outperforms

SVMs model since the predicted line of ANNs locates closer to actual line. The graphs are based on actual against predicted RR resulted from the training process for the period of January 2000 until December 2008.

Figure 3. Actual vs. Predicted Graph Analysis

This finding is also supported by accuracy rate calculation (eq.11) resulted from training process in which ANNs achieves 94.72% and SVMs achieves 92.93%. Furthermore, using statistical criteria as shown in table 3, it supports the graph analysis finding that performance of ANNs is better since the value of R2 is higher than such value of SVMs. On the other hand, value of MAE and NMSE of ANNs are lower than SVMs models. The all conditions mentioned above demonstrate that ANN outperforms SVMs models empirically in case of learning ability and explaining RR volatility.

TABLE 3. PERFORMANCE COMPARISON USING STATISTICAL CRITERION

Finally, the out of sample data prediction is carried out interestingly, out of sample prediction suggested that ANNs performs slightly less accurate than SVMs in which the average accuracy rates are 78.6% and 83.50% respectively.

IV. CONCLUSION From this experiment, we may conclude that ANNs can

be selected as a predictive algorithm due to its very good performance in learning and capturing data pattern which resulting an acceptable rate of accuracy. However, regarding the slightly less performance compared with SVMs in making out of sample prediction; it suggests that this is necessary to find out a better structure of networks , learning strategies and algorithm which are able to minimize more error rather than structural risk minimization principle. For future research, the ANNs will be employed as predictive algorithm in web-based decision support system for depositor of Islamic bank. The system hopefully will benefit not only for depositors, but also for Islamic bank industry in general. Because, the prediction information of RR will provide options for depositors to find other Islamic bank which probably provide better return. As a result, it will keep depositor funds stay longer in Islamic bank industry before shifting to its counterpart.

ACKNOWLEDGMENT This research is financed by Ministry of Education,

Republic of Indonesia. Saiful Anwar and Rifki Ismal thank to Prof. Dr. H. Fathurrahman Djamil from STIE Ahmad Dahlan, Jakarta for the very kind support.

REFERENCES [1] A.F. Atiya, “Bankruptcy Prediction for Credit Risk Using Neural

Networks: A survey and New results”, IEEE Transaction on Neural Networks, vol.12, no.4, pp. 929-935, 2001.

[2] M.A. Azis and H.A. Dar, “Predicting Corporate Bankruptcy: Whither do We Stand?”, working paper, Department of Economics, Loughborough University, UK, 2004, unpublished.

[3] M. Bashir, “Determinants of Profitability and Rate of Return Margins in Islamic Banks: Some Evidence from the Middle East”, Economic Research Forum, available at: http://www.erf.org.eg/CMS/getFile.php?id=753, Oct,2000.

[4] M.C. Bishop, Neural networks for pattern recognition 1st ed, Oxford University Press Inc, New York, 1995.

[5] P. Bourke, “Concentration and Other Determinants of Bank Profitability in Europa, North America and Australia”, Journal of Banking and Finance, vol.13, pp. 65-67, 1989.

[6] C.M, Gambs, “Bank Failures-An Historical Perspective”, Federal Reserve Bank of Kansas City Monthly Review, June, pp.10-20, 1977.

[7] Jirong. Gu, Zhu. Mingcang and Jiang. Liuguangyan, “Housing Price Forecasting Based on Genetic Algorithm and Support Vector Machine”, Expert System with Application, vol.38, pp.3383-3386, 2011.

[8] R. Ismal, “Industrial Analysis of Liquidity Risk Management in Islamic Bank”, Journal of Islamic and Finance, International Association of Islamic Bank, Karachi, Pakistan, vol.26, no. 2, 2009.

[9] I. Juhos, L. Makra and B. Toth, “Forecasting of Traffic Origin No and NO2 Concentration by support Vector Machine and Neural Networks Using Principal Component Analysis”, Simulation and Modelling Practice and Theory, vol 16, no.9, pp.1488-1502, 2003.

[10] Boyang. Li, Jinglu. Hu and K. Hirasawa, “Financial Time Series Prediction Using a Support Vector Regression Network”, International Joint Conference on Neural Networks (IJCNN), 2008.

[11] Chi-Jie. Lu, Tian-Shyung. Lee and Chih-Chou. Chiu, “Financial Time Series Forecasting Using Independent Component Analysis and

Statistical Criterion ANNs SVMs

R2 0.88 0.85 MAE 0.56 0.63

NMSE 0.19 0.40

ANNs MODEL

SVMs MODEL

260260

Support Vector Regression”, Decision Support System, vol.47, pp.115-125, 2009.

[12] N.A.B. Mazlina and M.T. Izah, “Applying Multiple Linear Regression and Neural Network to Predict Bank Performance”, International Business Research, vol 2, no 4, pp. 176-181, 2009.

[13] N. Nguyen and A. Cripps, “Predicting Housing Value: A Comparison of Multiple Linear Regression Analysis and Artificial Neural Networks”, Journal of Real Estate Research, vol.22, no.3, pp. 313-336, 2001.

[14] A.S. Nugroho, A.B. Witarto, and D. Handoko, “Support Vector Machine: Teori dan Aplikasinya dalam Bioinformatika”, Kuliah Umum IlmuKomputer.com, 2003.

[15] Ing-Feng. Pai , Chih-Shen. Lin, Wei-Chiang. Hong and Chen-Tung. Chen, “A Hybrid Support Vector Machine Regression for Exchange Rate Prediction”, Information and Management Sciences, vol.17. no.2, pp.19-32, 2006.

261261