Discussion

Comment on ‘Comparison of static-feedforward

and dynamic-feedback neural networks for rainfall-runoff

modeling’ by Y.M. Chiang, L.C. Chang, and F.J. Chang, 2004.

Journal of Hydrology 290 (3–4), 297–311

Ashu Jain*

Department of Civil Engineering, Indian Institute of Technology Kanpur, Kanpur, UP 208 016, India

Received 23 September 2004; revised 23 September 2004; accepted 24 March 2005

1. Introduction

The superiority of Artificial Neural Networks

(ANNs) over the traditional statistical and concep-

tual techniques in modeling the hydrological process

has been emphasized by many researchers in recent

years (Raman and Sunilkumar, 1995; Tokar and

Markus, 2000; Jain and Indurthy, 2003). In their

paper, Chiang et al. (2004) compare two back-

propagation (BP) training algorithms, namely

standard BP and conjugate gradient (CG), with the

real-time recurrent learning (RTRL) algorithm for

modeling hourly rainfall-runoff process in Lan-Yang

River watershed having an area of 978 km2 located

in Taiwan. The authors employed two statistical

parameters, mean absolute error (MAE) and root

mean square error (RMSE) to develop and test

various ANN models for different cases that

involved considering different storms in training,

0022-1694/$ - see front matter q 2005 Elsevier B.V. All rights reserved.

doi:10.1016/j.jhydrol.2005.03.037

* Tel.: C91 512 259 7411; fax: C91 512 259 7395.

E-mail address: ashujain@iitk.ac.in.

validation, and testing data sets. The authors

conclude that CG performs better than the standard

BP method and RTRL performs better than both the

BP methods particularly when the training data set

is limited. The work reported by the authors

represents an important stride in an ongoing

research effort of exploring the use of ANNs for

efficient modeling of the hydrological process. The

discusser would like to comment on some important

issues related to the ANN modeling of the rainfall-

runoff process and the advances made in the recent

past. The discusser would also like to comment on

certain considerations that need further clarification,

which would help readers in better understanding

and extending the authors’ work. The comment

begins by first making a few general observations

and citing the recent advancements in the area of

ANN modeling of the complex rainfall-runoff

process (Section 2). Specific issues related to the

work presented by Chiang et al. (2004) are

presented in Section 3 before making concluding

remarks.

Journal of Hydrology 314 (2005) 207–211

www.elsevier.com/locate/jhydrol

A. Jain / Journal of Hydrology 314 (2005) 207–211208

2. General observations relating to ANN modeling

of rainfall-runoff process

The complexity and non-linearity in the rainfall-

runoff process is mainly due to the complex and

non-linear storage characteristics of the watershed,

varying infiltration capacities, and interactions among

various components of the hydrologic process (Zhang

and Govindaraju, 2000). The large watersheds with

flatter overall slopes would involve higher degree of

complexity and non-linearity in the storage charac-

teristics and as such would be more difficult to model

using any technique. The watershed considered in the

study presented by the authors is small and steeply

sloped indicating that the complexity, dynamics, and

non-linearity involved in the rainfall-runoff process

would be of lesser degree. For such a watershed, it

would be relatively easier for a simpler ANN structure

to capture the rainfall-runoff process. Recently, there

has been a shift in the philosophy in developing

systems theoretic models such as ANN models.

Traditionally, such models are developed as black-

box models that do not consider the underlying

physics. However, ANN models can be developed as

gray-box models such that the details of the

underlying physics are partially visible. This can be

achieved by considering the time of concentration

(TC) of the watershed and deciding on the number of

rainfall input variables based upon TC (Jain and

Indurthy, 2004) or embedding the conceptual com-

ponents such as infiltration, evaporation, and soil

moisture accounting procedures into an overall ANN

model (Jain and Srinivasulu, 2004).

The training of an ANN is the most important

aspect in developing rainfall-runoff models. Most of

the ANN applications in hydrology employ the

standard BP training algorithm or its variants as

rightly pointed out by the authors. However, in the last

decade or so, it has been reported by many researchers

that the BP and its variants are not efficient in

capturing the rainfall-runoff relationships inherent in

varying magnitude events e.g. low, medium, and high

flows. Curry and Morgan (1997) reported that the use

of gradient search techniques, such as those employed

in the BP and its variants, often result in inconsistent

and unpredictable performance of the neural net-

works. Hsu et al. (1995) developed rainfall-runoff

models using ANNs and experienced that the ANN

models under-predicted low flows and over-predicted

medium flows. Hsu et al. (1995) concluded that this

may have been due to the ANN models not being able

to capture the non-linearity in the rainfall-runoff

process and suggested that there is still room for

improvement in applying different training algor-

ithms, such as stochastic global optimization and

genetic algorithms, to reach near global solutions, and

achieve better model performances. Ooyen and

Nichhuis (1992) tried to improve the convergence

during training of the ANNs using BP algorithm and

experienced that the convergence was slow and the

learning process was ‘inefficient’ when the output

data contained values near to zero or unity. Sajikumar

and Thandaveswara (1999) and Tokar and Markus

(2000) also experienced that the patterns with target

values in the neighborhood of zero (i.e. low flows)

will not be learnt properly by BP algorithm in ANN

rainfall-runoff models. Recently, Jain and Srinvasulu

(2004) have proposed the use of real-coded genetic

algorithms (RGA) for developing ANN rainfall-runoff

models that overcome many of these limitations. They

compared standard BP and RGA training algorithms

for developing ANN models of the complex rainfall-

runoff process in the Kentucky River watershed (area

more than 10,000 km2) and found that the ANN

models trained using RGA exhibited much superior

generalization capability in capturing the complex

rainfall-runoff relationships for varying magnitude

flows than the ANN models trained using BP method.

Another aspect that is very important in the area of

ANN rainfall-runoff modeling is the performance

evaluation of various ANN structures investigated

and the selection of the best model that can be

employed in important water resources management

activities. A model that is ‘efficient’ in capturing the

complex, dynamic, and highly non-linear rainfall–

rainfall process and ‘effective’ in accurately predicting

low, medium, and high magnitude flows is desirable.

Normally, only a few global statistical measures such

as correlation coefficient (R), Nash-Sutliffe efficiency

(E), and root mean square error (RMSE) etc. are

employed to evaluate the relative performances of

various ANN models investigated. These statistical

measures are good indicators of the ‘robustness’ or the

‘efficiency’ of the models in capturing the complex

relationships inherent in the input and output data.

However, such error statistics suffer from a few

A. Jain / Journal of Hydrology 314 (2005) 207–211 209

drawbacks: (a) they are biased towards high magnitude

flows due to the involvement of the sum of square of

the differences between the observed and modelled

flows, and (b) they are not good indicators of the

‘effectiveness’ in accurately estimating the varying

magnitude flows (Jain and Srinivasulu, 2004). There-

fore, different statistics that are unbiased in nature and

provide useful information about the ‘effectiveness’ in

accurately predicting the varying magnitude flows,

such as average absolute relative error (AARE),

threshold statistics (TS), and normalized mean bias

error (NMBE), etc. are needed (Jain et al., 2001; Jain

and Ormsbee, 2002; Jain and Srinivasulu, 2004).

Therefore, a wide variety of standard statistical

measures is needed to evaluate the ‘efficiency’ in

modeling and ‘effectiveness’ in accurately predicting

future flows to properly evaluate the performance of

various ANN models investigated and select the best

model to be incorporated in the important water

resources management activities. Also, the lead-time

of runoff forecasts is an important aspect to be

considered in developing ANN rainfall-runoff model

as the lead-time of the runoff forecasts plays very

important role in the water resources management

activities. Many water resources applications and flood

management activities require more than one-hour of

lead-time forecasts to plan flood warnings and to

implement evacuation plans. ANNs are robust models

capable of capturing the complex, dynamic, and highly

non-linear physical processes involving larger time

horizons. It would have been interesting and more useful

from practical considerations to explore the ANN

technique in forecasting streamflow at larger lead-times.

3. Specific comments

1. In Section 2.1, item 3, the authors mentioned that

‘if the validation data set indicates that the network is

over-trained, then the network should be retrained

using a different number of neurons and/or parameter

values’. Such statements can be confusing for the

potential users of ANNs as the role of the validation

set is to prevent the over-training or under-training by

stopping the training at a certain number of epochs at

which the global error during validation set is

the minimum. This raises another question about

the methods adopted by the authors for training all the

models for 10,000 epochs (more on this later).

2. The authors report that the static neural networks

(SNNs) have several drawbacks (a) they may fail to

produce satisfactory solution due to insufficient data

set, and (b) they may not cope well with changes that

were never learnt during training phase. The discusser

would like to point out that (a) there are studies that

demonstrate that the SNNs can produce satisfactory

results if developed properly even with very limited

data (Jain and Indurthy, 2004), and (b) if the training,

validation, and testing data sets are designed properly

taking care of possible ranges of future flows then the

SNNs are able to provide good results. Moreover,

SNNs are more suited in the important water resources

optimization and management models in which

trained ANN can be employed to produce future

flows in no time. The RTRL networks may require

time consuming step for making predictions of flows.

The discusser agrees with the authors that the SNNs

employing BP can easily fall into a local minima and

the speed of convergence of the network can be slow

when the number of input in the data set is large. The

standard BP and its variants do not guarantee global

minima and other training methods such as RGA need

to be employed to get better generalization capabili-

ties (Jain and Srinivasulu, 2004).

3. Figure 4 shows that the rain gauges selected are

not evenly distributed in the watershed. For example,

rain gauges R2, R3, and R4 are very close to each

other and their rainfall characteristics will definitely

be correlated. Therefore, in the opinion of the

discusser, the complexity of the ANN models could

have been reduced by taking only two rain gauges

(e.g. R1 and R3). Alternatively, it could have been

worthwhile to develop lumped ANN models by

considering spatially averaged rainfall (arithmetic or

Thissen-Polygon) in the input vector. The latter

approach offers the advantage of having simpler

ANN structure due to fewer neurons in the input layer

when the performance of the lumped model is either

comparable or even slightly worse than the complex

ANN models developed by considering distributed

rainfall data as done by the authors.

4. It appears that much effort was devoted to

developing various ANN models. Firstly, a lot of

effort has been expended in developing four models

that are different in the lag times for past flows and

rainfalls considered, which could have been avoided.

The time of concentration in the watershed can give

A. Jain / Journal of Hydrology 314 (2005) 207–211210

very useful information about the number of time

steps in the past for which rainfall needs to be

considered as input neurons (Jain and Indurthy, 2003).

Also, guidelines for developing ANNs suggest that

auto-correlation functions (ACF), partial auto-

correlation functions (PACF) and cross-correlation

functions (CCF) can be used to decide on the number

of significant explanatory variables to be considered

in the ANN model (Sudheer et al., 2002; Jain and

Srinivasulu, 2004) rather than attempting a trial and

error procedure of employing all the data available, as

done by the authors. Secondly, it appears that the

training of all the ANN models developed by the

authors is questionable as it did not concentrate on

obtaining optimum ANN structure for each type of

model and case considered resulting in either under-

trained or over-trained networks. For all the ANN

structures, the number of hidden neurons was varied

from 1 to 15 and all the ANN structures were trained

for 10,000 iterations. The discusser feels that training

all ANN structures of varying hidden neurons and

complexity for the same number of epochs is not

optimal. Normally, an ANN is trained until a certain

level of acceptable error is achieved. A close look at

Figure 5 indicates that 3, 5, 10, and 11 hidden neurons

are more appropriate for models 1, 2, 3, and 4,

respectively. Adoption of an approach followed by the

authors would lead to either under-training or over-

training of various ANN structures investigated,

which is evident in the results presented. For example

(a) RMSE values during training and validation from

models 1 and 2 (Table 2) indicate under-training (e.g.

RMSE for models 2 and 3) that may have been due to

the higher number of hidden neurons selected when

only fewer may have been sufficient, (b) MAE and

RMSE statistics during training, validation, and

testing (see Table 4) for both CG and RTRL indicates

over-training (RMSE of 110 for RTRL during training

jumps by more than 50% to 166 during testing), (c)

the results from Table 5 show the RMSE value (79)

during training increases by 185% during testing

(225) for CG and by 135% from 96 to 225 for RTRL,

and (d) the results from Table 7 show the RMSE value

(70) during training increases by 256% during testing

(249) for CG and MAE value (40) during training

increased by 183% during testing (113) for CG. Also,

the statements made by the authors in the conclusions

such as ‘it appears the accuracy in the testing phase

was inferior even though a more accurate result is

obtained in the training phase’ and ‘examining the

forecasted results of the CG method for cases 1–4, it

can be seen that the CG method produces much less

accurate results in the test phase than the training

phase for all the cases except case-1’ clearly indicate

over-trained ANNs.

5. The scatter plots presented in Figure 10 are for

training, validation, or testing phase is not clear.

Further, all the scatter plots clearly indicate that the

ANN models for all the cases have not been able to

estimate high magnitude flows accurately as the

spread from the ideal line increases with the

magnitude. This suggests the inferior generalization

capability of the training methods adopted (BP, CG,

and RTRL) in developing various ANN models,

which can be overcome by the use of RGA.

6. The authors have carried out a limited evaluation

of the developed ANN models using only two

statistical performance evaluation criteria (MAE and

RMSE). It has been pointed out by many researchers

that a wide variety of performance evaluation

statistics, such as R, E, AARE, TS, NMBE, and

RMSE, etc. is needed to evaluate the performance of

the ANN models and conclude about the ‘efficiency’

and ‘effectiveness’ of one model over the other.

7. The discusser does not agree by the authors’

statement that since the model 3 produced the best

results, the average lag time is no more than three

hours. The time of concentration is a deterministic

characteristic of the watershed that needs to be

estimated before attempting to model the rainfall-

runoff relationship in the watershed. The results

merely indicate that the model-3 fits best to the data

available and conclusion regarding time of concen-

trations etc. cannot be made by looking at the limited

evaluation of ANN models carried out by the authors

that are poorly trained.

8. The authors have provided some details of the

ANN models trained using BP and CG methods but

the procedures of developing RTRL models are not

clearly explained. For example, why only five hidden

neurons were selected in the ANN models to be trained

using RTRL? Further, the authors have concluded that

RTRL models are better than the BP or CG models;

however, such conclusions are not supported by

the results presented. For example, the results

presented in Table 4 indicate that the CG model is

A. Jain / Journal of Hydrology 314 (2005) 207–211 211

far superior to the RTRL model in terms of both MAE

and RMSE for case-1 that involves more data. Why

the RTRL method fails to produce better results when

presented with more data? The discusser feels that it

may be due to poorly trained ANN models. Further,

the RTRL models cannot be used in a water resources

operation and management activity based on the sole

observation of them being able to produce peak

discharge with a slightly better accuracy as mentioned

by the authors in the conclusions section.

4. Conclusions

The research work presented by the authors is an

important step in developing efficient models of the

hydrological process using ANNs. A response from

the authors to clarify certain issues raised in this

comment would not only help in better understanding

of the authors’ work but also increase the utility of the

authors’ work.

References

Chiang, Y.M., Chang, L.C., Chang, F.J., 2004. Comparison of static

feedforward and dynamic feedback neural networks for rainfall-

runoff modeling. J. Hydrol. 290 (3–4), 297–311.

Curry, B., Morgan, P., 1997. Neural network: a need for caution.

Omega, Int. J. Manage. Sci. 25 (1), 123–133.

Hsu, K.-L., Gupta, H.V., Sorooshian, S., 1995. Artificial neural

network modeling of the rainfall-runoff process. Water Resour.

Res. 31 (10), 2517–2530.

Jain, A., Ormsbee, L.E., 2002. Evaluation of short-term water

demand forecast modeling techniques: conventional methods

versus AI. J. Am. Water Works Assoc. 94 (7), 64–72.

Jain, A., Indurthy, S.K.V.P., 2003. Comparative analysis of event

based rainfall-runoff modeling techniques-deterministic, statisti-

cal,andartificialneuralnetworks. J.Hydrol.Eng.,ASCE8(2),1–6.

Jain, A., Srinivasulu, S., 2004. Development of effective and

efficient rainfall-runoff models using integration of determinis-

tic, real-coded genetic algorithms, and artificial neural network

techniques. Water Resour. Res. 40 (4), W04302.

Jain, A., Indurthy, S.K.V.P., 2004. Closure of comparative

analysis of event based rainfall-runoff modeling techniques–

deterministic, statistical, and artificial neural networks. ASCE J.

Hydrol. Eng., 9(6), 551–553.

Jain, A., Varshney, A.K., Joshi, U.C., 2001. Short-term water

demand forecast modelling at IIT Kanpur using artificial neural

networks. Water Resour. Manage. 15 (5), 299–321.

Ooyen, A.V., Nichhuis, B., 1992. Improving convergence of back

propagation problem. Neural Networks 5, 465–471.

Raman, H., Sunilkumar, N., 1995. Multivariate modeling of water

resources time series using artificial neural network. J. Hydrol.

Sci. 40, 145–163.

Sajikumar,N.,Thandaveswara,B.S.,1999.Anon-linear rainfall-runoff

model using an artificial neural network. J. Hydrol. 216, 32–55.

Sudheer, K.P., Gosain, A.K., Ramasastri, K.S., 2002. A data-driven

algorithm for constructing artificial neural network rainfall-

runoff models. Hydrol. Processes 16 (6), 1325–1330.

Tokar, A.S., Markus, M., 2000. Precipitation runoff modeling using

artificial neural network and conceptual models. J. Hydrol. Eng.,

ASCE 5 (2), 156–161.

Zhang, B., Govindaraju, S., 2000. Prediction of watershed runoff

using Bayesian concepts and modular neural networks. Water

Resour. Res. 36 (3), 753–762.