Quality assurance in model based water management - review of existing practice and outline of new approaches

www.elsevier.com/locate/envsoft

Environmental Modelling & Software 20 (2005) 1201–1215

Quality assurance in model based water management – review ofexisting practice and outline of new approaches

Jens Christian Refsgaarda,), Hans Jørgen Henriksena, William G. Harrara,Huub Scholtenb, Ayalew Kassahunb

aGeological Survey of Denmark and Greenland (GEUS), Øster Voldgade 10, DK-1350 Copenhagen K, DenmarkbWageningen University (WU), Dreijenplein 2, 6703 HB, Wageningen, The Netherlands

Received 11 December 2003; received in revised form 30 March 2004; accepted 30 July 2004

Abstract

Quality assurance (QA) is defined as protocols and guidelines to support the proper application of models. In the watermanagement context we classify QA guidelines according to how much focus is put on the dialogue between the modeller and the

water manager as: (Type 1) Internal technical guidelines developed and used internally by the modeller’s organisation; (Type 2)Public technical guidelines developed in a public consensus building process; and (Type 3) Public interactive guidelines developed aspublic guidelines to promote and regulate the interaction between the modeller and the water manager throughout the modellingprocess. State-of-the-art QA practices vary considerably between different modelling domains and countries. It is suggested that

these differences can be explained by the scientific maturity of the underlying discipline and differences in modelling markets in termsof volume of jobs outsourced and level of competition. The structure and key aspects of new generic guidelines and a set ofelectronically based supporting tools that are under development within the HarmoniQuA project are presented. Model credibility

can be enhanced by a proper modeller-manager dialogue, rigorous validation tests against independent data, uncertaintyassessments, and peer reviews of a model at various stages throughout its development.� 2004 Elsevier Ltd. All rights reserved.

Keywords: Modelling guidelines; Quality assurance; Water resources management; Uncertainty; Support tools

1. Introduction

Models describing water flows, water quality andecology are being developed and applied in increasingnumber and variety. The trend in recent years has beento base water management decisions to a larger extenton modelling studies, and to use more sophisticatedmodels. In Europe this trend is likely to be reinforced bythe EU Water Framework Directive due to its demandfor integrating groundwater, surface water, ecological

) Corresponding author. Tel.: C45 38 142 776; fax: C45 38 142

050.

E-mail address: [email protected] (J.C. Refsgaard).

1364-8152/$ - see front matter � 2004 Elsevier Ltd. All rights reserved.

doi:10.1016/j.envsoft.2004.07.006

and economic aspects of water management at the riverbasin scale and due to the explicit requirement to studyimpacts of alternative measures (human interventions)intended to improve the ecological status in the riverbasin. Insufficient attention is often given to document-ing the predictive capability of models. Therefore,contradictions may emerge regarding the various claimsof model applicability on the one hand and the lack ofdocumentation of these claims on the other hand.Hence, the credibility of the model is often questioned,and sometimes with good reason.

Another important trend is the demand to involvedifferent stakeholders in the water resources manage-ment process, and therefore also indirectly in themodelling process (Pahl-Wostl, 2002). This stakeholder

mailto:[email protected]

http://www.elsevier.com/locate/envsoft

1202 J.C. Refsgaard et al. / Environmental Modelling & Software 20 (2005) 1201–1215

involvement does not imply active participation inthe technical modelling itself, but rather appears asa demand to be able to understand and review thevarious assumptions and their implications for themodelling results. This trend is seen at the global scalein connection with the generally accepted principlesbehind integrated water resources management, wherepublic participation is a key element (GWP-TAC, 2000).In Europe, this is reflected in the EU Water FrameworkDirective, where it is explicitly prescribed that stake-holders and the general public should be involved in thewater resources management process.

The need for improving the quality of the modellingprocess has been emphasised by the research com-munity, e.g. Klemes (1986), NRC (1990), Anderson andWoessner (1992), Forkel (1996), and Rykiel (1996). Therecommendations made in this respect primarily focuson scientific/technical guidance on how the modellershould carry out various steps during the modellingprocess in order to achieve the best and most reliableresults.

Anderson and Bates (2001) in a discussion of modelcredibility and scientific integrity state that ‘‘over the lastdecade we have begun to have an appreciation of theneed to be much more rigorous in establishingprocedures for defining model credibility’’. They arguefurther that this demand has not evolved from thehydrological science itself due to immaturity and datalimitations, but instead comes from policy makers andregulators who wish to have some kind of certificationof model results.

As emphasised by e.g. Forkel (1996) modellingstudies involve several partners with different responsi-bilities. The ‘key players’ are code developers, modelusers and water managers. However, a lack of mutualunderstanding may develop due to the complexity of themodelling process and the different backgrounds of the‘key players’. For example, the strengths and limitationsof modelling applications are often difficult, if notimpossible, for the water managers to assess. Similarly,the transformation of objectives defined by the watermanager to specific performance criteria can be verydifficult for the model users to assess. It can be difficultto audit modelling projects due to the lack of properdocumentation and transparency. Furthermore, it isoften difficult to reconstruct and reproduce the model-ling process and its results.

In the water resources management community manydifferent guidelines on good modelling practise havebeen developed. One of, if not the most, comprehensiveexample of a modelling guideline has been developed inThe Netherlands (Van Waveren et al., 2000; Scholtenand Groot, 2002) as a result of a process involving allthe main players in the Dutch water management field.The background for this process was a perceived needfor improving the quality in modelling by addressing

malpractice issues such as careless handling of inputdata, insufficient calibration and validation, and modeluse outside its intended scope (Scholten et al., 2000).Similarly, modelling guidelines for the Murray-DarlingBasin in Australia were developed due to the perceptionamong end-users that model capabilities may have been‘over-sold’, and that there was a lack of consistency inapproaches, communication and understanding amongand between the modellers and the water managers,which often resulted in considerable uncertainty fordecision making (Middlemis, 2000).

As pointed out by Merrick et al. (2002) goodmodelling practice cannot be decomposed into a set ofrigid rules that can be followed without communicationbetween modellers and water managers. Furthermore,there is a risk that modellers will not embrace guidelinesaiming to inject too much consistency in the reviewprocedure. Experiences from Australia have shown thatreview reports are commonly interpreted by watermanagers (non-modellers) as quite negative. Non-modellers may tend to focus mainly on the negativereview comments rather than balance those against thepositive comments. This may mostly be the case forprojects where there has not been a proper specificationof the purpose and conditions at the initiation of themodel study or where previous reviews during earlierproject stages have been inadequate. External reviewsperformed at the end of a project when things may havealready gone wrong may often result in defensiveresponses both from the modellers and the watermanagers (Henriksen, 2002a).

All the existing modelling guidelines that we areaware of exist as reports. Electronically based support isonly available as text forms to record modellingactivities. No electronically based tool that is coupledto a knowledge base defining how to carry out themodelling (electronic version of guidelines with com-prehensive guidance to different types of users) exists atpresent. This is a paradox, considering the significantresources that are invested in improving modellingsoftware packages with respect to new sophisticatedinformation technology.

Poor modelling results may be caused by the lack ofadequate model codes, or data of insufficient quantity orquality. However, according to our experience the mostprevalent reason for poor modelling results is theinadequate use of guidelines and quality assuranceprocedures, and improper interaction between themanager (client) and the modeller (consultant). Ourwork has been carried out within the context of an EUsupported research project (http://www.harmoni-qua.org) aimed at developing a common set of qualityassurance guidelines and supporting software tools. Thescientific philosophical basis for the adopted terminol-ogy and guiding principles are described by Refsgaardand Henriksen (2004). The objective of the present

http://www.harmoniqua.org


1203J.C. Refsgaard et al. / Environmental Modelling & Software 20 (2005) 1201–1215

paper is to establish new approaches and outline therequirements of supporting tools for quality assuranceprocedures in the modelling process.

2. Theoretical framework

2.1. Terminology and scientific basis

The terminology and methodology used in thefollowing are based on Refsgaard and Henriksen (2004).The key elements in the terminology are illustrated inFig. 1 and the most important definitions are:

� A model code is a generic software program, whichcan be used for different study areas withoutmodifying the source code.

� A model is a site application of a code to a particularstudy area, including input data and parametervalues.

� A model code can be verified. A code verificationinvolves comparison of the numerical solutiongenerated by the code with one or more analyticalsolutions or with other numerical solutions. Verifi-cation ensures that the computer programme accu-rately solves the equations that constitute themathematical model.

� Model validation is here defined as the process ofdemonstrating that a given site-specific model iscapable of making accurate predictions for periodsoutside a calibration period. A model is said to bevalidated if its accuracy and predictive capability inthe validation period have been proven to lie withinacceptable limits or errors.

These terms are commonly used, although withdifferences in meaning between authors. Our views on

Fig. 1. Elements of a modelling terminology (Refsgaard and

Henriksen, 2004).

these terms and the ongoing discussion on validation-falsification-confirmation as well as between the termsperceptual model, conceptual model and site-specificmodel are given in Refsgaard and Henriksen (2004).Here we just note that, from a quality assuranceguideline point of view, it is fundamental for us tomake a clear distinction between the terms conceptualmodel, model code and (site-specific) model. Further-more, we never use the terms verification and validationin a universal sense, but always restricted to clearlydefined domains of applicability (numerical universal inPopperian sense).

In addition to ensure a proper quality of work thethree most important underlying principles that havebeen identified from an analysis of the modelling processare (Refsgaard and Henriksen, 2004):

� Validation tests against independent data that havenot also been used for calibration are necessary inorder to be able to document the predictivecapability of a model.

� Model predictions achieved through simulationshould be associated with uncertainty assessmentswhere amongst others the uncertainty in modelstructure and parameter values should be accountedfor.

� A continuous interaction between water manager andmodeller is crucial for the success of the modellingprocess. One of the key aspects in this regard is toestablish suitable performance criteria for the modelcalibration and validation tests. This dialogue is alsovery important in connection with uncertaintyassessments.

2.2. Types of QA guidelines

2.2.1. Definition and classificationof quality assurance (QA)

Quality assurance (QA) is defined by NRC (1990) asthe procedural and operational framework used by anorganisation managing the modelling study to assuretechnically and scientifically adequate execution of alltasks included in the study, and to assure that allmodelling-based analysis is reproducible and defensible.In line with this we define QA guidelines as protocolsand guidelines to support good application of models inwater management.

QA in the modelling process has two main compo-nents: (a) QA in development of model codes; and (b)QA in relation to application studies. Our paper focuseson the second component only.

QA in model application studies includes dataanalyses, methodologies of good modelling practice,reviews and administrative procedures. Such QA guide-lines can be classified according to how much focus is


put on the consensus building process between themodeller and the water manager in the following threeclasses:

� Internal technical guidelines (Type 1) established andused internally by the modeller’s organisation.

� Public technical guidelines (Type 2) established aspublic guidelines and used internally by the model-ler’s organisation.

� Public interactive guidelines (Type 3) established aspublic guidelines and based on regulation of theinteraction between the modeller and the watermanager throughout the modelling process.

2.2.2. Type 1: Internal technical guidelinesMost organisations involved in modelling studies

have some kind of internal QA procedures. They usuallyfocus on the technical aspects, i.e. to ensure that themodelling work itself is done without making un-qualified judgements or errors. The betters of these arebased on the modelling protocols and similar scientif-ically based procedures originating from the researchcommunity. These procedures are internal in naturebecause they have been established or adopted unilat-erally by the modeller’s organisation, and because theyseldom deal with the interaction between modeller andend-user. Examples of Type 1 guidelines include:

� Internal QA procedures, common in many compa-nies.

� Text books. Many textbooks contain chapters withrecommended modelling protocols (e.g. Andersonet al., 1993).

� Manuals to software packages with hints on the bestway to use a model (e.g. Rumbaugh and Rumbaugh,2001; DHI, 2002).

2.2.3. Type 2: Public technical guidelinesThese guidelines often contain the same substance as

the internal technical guides mentioned above. How-ever, they differ in the sense that they have beenprepared through a consultative and consensus buildingprocess involving many persons and organisations. Theyfocus on the technical aspects and give no or littleemphasis to the interaction between the modeller andthe end-user. Examples of Type 2 guidelines include:

� The CAMASE guidelines for modelling that weredeveloped after substantial consultation within thescientific modelling community (CAMASE, 1996).

� Standards from American Society for Testing andMaterials (e.g. ASTM, 1994).

� Many of the UK standards, especially the older ones(Packman, 2002).

2.2.4. Type 3: Public interactive guidelinesThese guidelines have, like the public technical

guidelines (Type 2), been established through a publicconsultative and consensus building process. However,they differ from the Type 2 guidelines by an additionalfocus on regulating the interaction between the modellerand the water manager, who often have the roles ofconsultant and client, respectively.

Important elements in public interactive guidelinesare reviews that, in addition to QA in the sense of tech-nical guidance, can facilitate the consensus-building pro-cess between the parties. Experience shows that such aprocess is crucial for the overall credibility of the model-ling process. Examples of such QA guidelines include(more details on these guidelines provided in nextchapter):

� The Dutch guidelines (Van Waveren et al., 2000;Scholten and Groot, 2002).

� The Australian groundwater flow modelling guide-lines established by the Murray-Darling BasinCommission (Middlemis, 2000; Merrick et al.,2002; Henriksen, 2002a).

� The Danish groundwater modelling guidelines(Henriksen, 2002b).

� Some of the recent UK standards (Packman, 2002).� Californian guidelines prepared by Bay-Delta Mod-elling Forum (BDMF, 2000).

2.3. Development stage and prevalenceof QA guidelines

Reviews of a number of existing QA guidelines (seedetails in next chapter) revealed significant differences incurrent practice, both between domains and betweendifferent countries. In some domains and some countriesthere has been a clear trend over the past couple ofdecades to move from Type 1 to Type 2 or Type 3guidelines. In order to understand the development ofQA guidelines and be able to provide recommendationsbased on anticipated future needs, it is important to tryto understand why the present differences in thedevelopmental stage of QA guidelines exist. Thehypothesis that we will test is that the developmentstage depends on two main factors:

� The scientific maturity of the underlying discipline,i.e. how well understood are the underlying pro-cesses and how easily available are the datanecessary for practical applications. In this respect,a mature scientific discipline is one where there isa general acceptance in the scientific community onhow the processes are described, there are nosignificant controversies on key issues, and it isfeasible to acquire the necessary data for practical


studies. Similarly, an immature scientific discipline isone where some processes are not well understood,where there are several alternative ‘schools’ on howto describe things, and where it is often not possibleto obtain sufficient field data necessary to performscientifically sound modelling. Immature scientificdisciplines are often considered as being complex,and are characterised by unresolved problems suchas scale problems. For example, whereas biology isa relatively old science in comparison with hydro-geology, biota (ecological) modelling is consideredto be immature in contrast to groundwater flowmodelling which is considered to be mature. Biotamodelling is rather uncertain due to the inherentcomplexity of ecological systems and the generallimited availability of relevant field data, whereas themathematical principles describing groundwaterflow are well established and flow systems arereadily characterised in the field.

� The modelling market maturity, i.e. how well de-veloped is the market for modelling studies. In thisrespect, a mature market is characterised by (a) themodelling market is relatively old with numerousexamples of good and poor quality modellingstudies, and the motivation for establishing QAguidelines is largely due to water managers havingexperience with studies of poor quality; (b) most jobsare outsourced to private consultants; (c) the volumeof modelling work is large, so that a number ofconsultants can be sustained and standard routinescan evolve; and (d) there is a considerable competi-tion among modellers in getting the jobs. Similarly,an immature market is characterised by (a) it is rela-tively new (typically !10 years); (b) most modellingstudies are carried out by government agencies them-selves; (c) the volume of work for the consultants issmall; and (d) there is virtually no competition

among modellers, instead the work is carried out bya few specialised groups which are often located in orhave close ties to the research community.

If these hypotheses were true one would a prioriexpect that a considerable degree of scientific maturity isrequired for QA guidelines of Type 2 to develop, andthat further a mature modelling market is a necessaryprerequisite for the development of Type 3 guidelines.

3. Existing guidelines

Reviews of existing QA guidelines were conducted(Refsgaard, 2002). The reviews attempted to cover twoaspects: (a) variation of practices between seven differentmodelling domains (groundwater, precipitation-runoff,hydrodynamics, flood forecasting, surface water quality,biota (ecology) and socio-economy); and (b) differencesbetween geographical regions. The reviews of state-of-the-art in the seven domains were carried out byseven different organisations with special expertise in therespective domains. During these reviews a broad searchof relevant QA guidelines were made with primary focuson existing guidelines in Europe and secondarilyon guidelines from North America and Australia.Subsequently, a few cases with guidelines from differentgeographical areas were selected for a more detailedreview. The reviews did not intend to be exhaustive byincluding all important QA guidelines, but aimed atselecting guidelines representative for conditions inEurope, North America and Australia.

In order to test the above hypotheses the conclusionsof the state-of-the-art of QA guidelines for the differentdomains summarised in Section 3.1 are plotted in Fig. 2as a function of scientific maturity. Furthermore,examples of guidelines from different countries are

QA

guidelines

Scientific

maturity

Mature

Immature

Type 2

Public

Type 3

Interactive

Type 1

Internal

GW-HD

GW-WQ

GW-AD

Modelling domainsGW-HD: Groundwater flowGW-AD: Groundwater solute transportGW-WQ: Groundwater geochemistryPR: Precipitation runoffHD: Hydrodynamic – surface water flowHD-Sed: Sediment transport/morphologyFF: Flood forecastingSWQ: Surface water qualityBiota: Biota (ecology)SE: Socio-economy

PR

HD

HD-Sed

FF

SWQ

BiotaSE

Fig. 2. State-of-the-art for QA guidelines in different modelling domains plotted against maturity of the underlying scientific disciplines.


QA

guidelines

Modelling

market

Mature

(Old, big,competive)

Immature

(New, small,specialised)

Type 2

Public

Type 3

Interactive

Type 1

Internal

BDMF

AUS-GW

NL-GMP

DK-GW

UK

Cases-guidelinesBDMF: Bay Delta Modelling Forum (California)AUS-GW: Australia, groundwaterNL-GMP: Dutch Good Modelling PractiseDK-GW: Denmark, groundwaterUK: United Kingdom, several domainsASTM: American Society for Testing and MaterialsCEE: Central and Eastern EuropeFR-FF: France, flood forecasting

ASTM

UK

CEE

FR-FF

UK

Fig. 3. Different types of guidelines as a function of maturity in the modelling market.

presented in Section 3.2 and Fig. 3 with focus on marketmaturity.

3.1. State-of-the-art in different modelling domains

Groundwater modelling (Refsgaard and Henriksen,2002): In this field, QA guidelines are well developedand used in many countries, but mostly in groundwaterflow modelling, where the state-of-the-art correspondsto Type 3 guidelines. For solute transport, and inparticular for geochemical modelling, relatively fewguidelines exist and they are not commonly used. Theneed for QA guidelines differs from country to country,amongst others due to different stages of development ofthe groundwater modelling market. For instance, theguides from the American Society for Testing andMaterials (ASTM) were among the first of their kind tobe developed, in the early 1990s, because the practicalapplication of groundwater models at that time hadprogressed further in the USA than in most othercountries.

Precipitation-runoff modelling (Perrin et al., 2002a):Relatively few guidelines exist for this domain as stand-alone guidelines. The guidelines that do exist are gen-erally confined to relatively simple (lumped) approaches,while no generic guidelines exist for the more complexmodels of the distributed physically-based type. Thus,the state-of-the-art for precipitation-runoff as a stand-alone domain may be characterised as Type1/Type2.However, it is also noted that precipitation-runoffmodelling is often used as an integral part of otherdomains, e.g. groundwater models, hydrodynamicmodels, flood forecasting models and surface waterquality models. For some of these integrated applica-tions some guidelines have been developed whichinclude the precipitation-runoff domain. This is, forinstance, the case for the Danish groundwater guidelines

(Henriksen, 2002b) which include aspects of precipita-tion-runoff modelling.

Hydrodynamic modelling (Metelka and Krejcik,2002a): This domain includes environmental applica-tions such as modelling of urban drainage and sewersystems, rivers, floodplains, estuaries and coastal watersboth with respect to flows, sediment and morphologicalissues. QA guidelines are well developed in some fields(e.g. in urban drainage and river modelling), but not inother fields (e.g. sediment and morphological model-ling). For hydrodynamic modelling in coastal areas andestuaries few QA guidelines have been identified. Thestate-of-the-art may be characterised as Type 2 for mostparts of the domain and Type 1 for other parts. It isnoted that hydrodynamic modelling is often an integralpart of flood forecasting and surface water qualitymodelling. Although very similar in theoretical scientificbackground, this domain is different from the field ofComputational Fluid Dynamics that typically is used forindustrial purposes.

Flood forecasting modelling (Balint, 2002): Thisdomain differs fundamentally from the other domainsby being based on real-time operation. This implies thatthe models, once established, are applied on a routine(daily) basis although often under extreme boundaryconditions. The focus on QA in this domain is oftenconcentrated on data quality for the on-line dataacquisition. Due to this fundamental difference in nature,the status of QA guidelines for this domain does not fitwell into the above classification, and it is not easilycomparable to the status of the other domains.

Surface water quality modelling (Da Silva et al.,2002): Surface water quality modelling is based ona description of physical, chemical and biologicalprocesses. Often the data availability to assess modelprocesses and parameters is sparse and often thekey processes are not well understood. QA guidelines


are generally not well developed. The state-of-the-artmay be characterised as Type 1.

Biota (ecological) modelling (Old et al., 2002):Ecology is a diverse branch of biology that focuses onthe relations of flora and fauna to one another and totheir physical environment. Ecological models arewidely used today, but perceived as being ratheruncertain due to the inherent complexity of ecologicalsystems and the general limited availability of relevantfield data. QA guidelines are generally not welldeveloped. The state-of-the-art may be characterised asType 1.

Socio-economic modelling (Heinz and Eberle, 2002):No general QA guidelines exist for socio-economicmodelling. The few existing guidelines, such as theCAMS, CFMPS and RBMPs in the UK, are specific forparticular types of application, and they are so far onlyused in practice in a few countries. The state-of-the-artmay be characterised as Type1/Type2.

In Fig. 2 the state-of-the-art for QA guidelines in therespective modelling domains have been plotted againstthe scientific maturity of the underlying disciplines. Thescientific maturity of the respective domains has beenassessed subjectively on the basis of the criteria outlinedin Section 2.3 above. There is a tendency that the leastdeveloped guidelines (Type 1) appear in domains wherethe underlying scientific basis is characterised asimmature, i.e. in surface water quality, biota (ecology)and groundwater quality, reflecting that many funda-mental scientific issues remain to be solved. Similarly,the Type 2 and Type 3 guidelines are dominant indomains characterised by scientific maturity. However,there are clear exceptions such as precipitation-runoffand flood forecasting, where other factors than scientificmaturity must play a role for the development stage ofQA guidelines.

3.2. Current practice in different countries

The current practice of using QA guidelines indifferent countries has been illustrated through someselected cases that have been reviewed in Refsgaard(2002). In Fig. 3 the type of QA guidelines used in thecase studies is plotted against the maturity of themodelling market that has been assessed subjectively onthe basis of the criteria given in Section 2.3 above. Thepractice as reflected by the case studies and shown onthe figure is summarised as follows:

Dutch guidelines (Scholten and Groot, 2002): TheDutch guidelines are the most generic of the existingguidelines in the sense that they cover all the domainsrelevant for river basin management. The technicalguidance for different modelling domains exist, but arenot as detailed as some of the guidelines that only coverone domain (e.g. ASTM guides or Australian guidelineson groundwater flow modelling). The Dutch guidelines

emphasise the dialogue process between modeller andwater manager, including the review procedures. TheDutch guidelines belong to Type 3. The Dutchmodelling market may be characterised as mature.

Australian groundwater flow modelling guidelines(Henriksen, 2002a): The Australian guidelines aretechnically comprehensive. They focus on the dialoguebetween the modeller and the water manager in generaland on review procedures in particular. The guidelineswere developed over several years with involvement ofall of the key stakeholders. The Australian guidelinesbelong to Type 3. The Australian groundwater model-ling market may be characterised as mature.

Danish groundwater modelling guidelines (Henriksen,2002b): The Danish Handbook of Good ModellingPractice and draft guidelines is similar to the Australianones, although some important details differ. The watermanagers, who also ensure that they presently are beingused in most studies, have initiated the Danish guide-lines. The Danish guidelines belong to Type 3. TheDanish groundwater modelling market may be charac-terised as mature.

Central and Eastern Europe (Metelka and Krejcik,2002b;Van Gils and Groot, 2002): Public QA guidelinesare neither well developed nor used. Many modellerstherefore rely only on internal QA procedures (Type 1)adopted by their respective organisations. This situationreflects a new and unregulated market for modellingservices, and a market where the managers and theirorganisations often are technically too weak to adoptand enforce QA guidelines.

French guidelines in flood forecasting (Perrin et al.,2002b): Public or interactive guidelines do not exist inthis area, and the case study describes a set of internaltechnical guidelines (Type 1). Although flood fore-casting is an old modelling discipline, the modellingmarket is virtually non-existent, because flood fore-casting modelling in France (as well as in most othercountries) is carried out either by a government agencyor by a specialised research institute.

UK guidelines (Packman, 2002): QA guidelines aregenerally very well developed in the UK. Application ofguidelines is prescribed as a routine in most areas ofmodel application. Thus, in general the UK market formodelling services is well regulated and characterised asbeing mature. Most of the guidelines are of Type 2 andsome recent ones of Type 3. The exceptions to this arethe surface water quality and biota (ecological) domainswhere no general guidelines exist. The guidelines in thesedomains are therefore confined to internal proceduresinspired by textbooks and manuals (Type 1).

Bay Delta Modelling Forum, California (BDMF,2000): The Californian guidelines provide a framework,but very few technical details. The main emphasis ofthese guidelines is on the interaction between modellers,managers and the public (Type 3). In this respect various


kinds of reviews are prescribed at various stages of themodelling process. The American market in general andthe Californian in particular are well established(mature).

American Society for Testing and Materials (ASTM,1992, 1994): The American guidelines are especiallycomprehensive in the groundwater domain, where theyhave served as inspiration for all the other groundwaterguidelines, including the Australian and the Danishguidelines. There are a number of guidelines on variouselements of the modelling process. These guides are 5–10years old and are mainly technical of nature, whilelimited focus is put on the interaction and reviewprocess.

In addition to the above QA guidelines ISO (theInternational Organisation for Standardisation) regu-larly publishes quality management and quality assur-ance standards. ISO standards provide guidance onfundamental principles and procedures, but on a rathergeneral level. We have found ISO standards addressingdevelopment, supply and maintenance of computersoftware (ISO 9000-3:1997) and other standards pro-viding guidance for a general process based qualitymanagement system in an organisation (ISO9004:2000(E)). However, none of the ISO standardsinclude any particular guidance on matters related towater resources modelling or management, and they aretherefore of limited practical use as compared to theabove other QA guidelines dedicated to water resourcesmodelling.

3.3. Content of existing guidelines

3.3.1. Key elementsThe existing guidelines all comprise modelling pro-

tocols with recommended steps and technical guidanceon how to perform these steps in the modelling process.The key elements may be divided into two groups,namely: (1) technical guides on how to use models; and(2) guides for regulating the interaction betweenmodeller and end-user/water manager. The key elementsin the technical guides include:

� Definition of the purpose of the modelling study.� Collection and processing of data.� Establishment of a conceptual model.� Selection of code or alternatively programming andverification of code.

� Model set-up.� Establishment of performance criteria.� Model calibration.� Model validation.� Uncertainty assessments.� Simulation with model application for a specificpurpose.

� Reporting.

The key elements in the interaction between themodeller and the end-user in addition to some of theabove elements also includes other aspects:

� Definition of the purpose of the modelling study,including translation of the end-users needs topreliminary performance criteria.

� Establishment of performance criteria. The accuracyof the model predictions has to be established viaa trade off between the benefits of improving theaccuracy in terms of less uncertainty on themanagement decisions and the costs of improvingthe accuracy through additional model studies and/or collection of additional field data.

� Reviews with subsequent consultation between themodeller and the end-user at different phases of themodelling project.

The content of the technical guides are to a largeextent domain specific, while the elements of theinteraction between the modeller and the end-user aremore general in nature and differ only slightly from onedomain to another.

3.3.2. Integration across modelling domainsAlmost all the existing guidelines were developed for

a specific domain e.g. groundwater modelling. Asintegrated modelling may be expected to play animportant role in connection with implementation ofthe EU Water Framework Directive and adoption ofIntegrated Water Resources Management principles,guidelines not including integrated modelling aspects areinadequate. Even the Dutch guidelines (Scholten andGroot, 2002) which cover a large number of domains areessentially single domain guidelines, because they do notprovide guidance on how to integrate across domains(interdependencies etc.). However, the Dutch guidelinesdo have the clear advantage over other existing guide-lines in that they are based on a common methodologyand a common glossary.

It should be noted though that some guidelines covermore than one modelling domain, as they are definedhere. For instance hydrodynamic modelling or ground-water modelling are often combined with precipitation-runoff, and guidelines combining these domains exist.

3.3.3. Differences in terminologyAs illustrated in Refsgaard (2002) the terminology

used in the modelling community varies significantlybetween domains and even to some extent from onecountry to another. This clearly demonstrates the needfor establishing one common terminology and glossaryfor modelling applications as addressed by Refsgaardand Henriksen (2004).


4. Outline of new guidelines – HarmoniQuA

4.1. Overall aim and structure

On the basis of the knowledge achieved through thereview of existing guidelines, the HarmoniQuA projectaims to develop a new comprehensive set of guidelinesand supporting software tools to facilitate an improvedquality of the modelling process and hence enhance theconfidence of all stakeholders.

HarmoniQuA forms part of the CATCHMODcluster of EU research projects (Blind, 2004). It aimsto be a methodological component of a future in-frastructure for model based decision support for watermanagement at catchment and river basin scale. Thismain goal will be reached by providing the elements ofa methodological layer in this infrastructure, embodiedin a knowledge base (KB) and software tools. Harmo-niQuA will collect methodological expertise, structurethis knowledge and identify and fill in gaps. It willconsist of generic and domain specific knowledge,modelling software specific aspects, and a transparentand consistent glossary of terms and concepts. Thisbody of knowledge will be structured in a knowledgebase. The following set of software tools will providefunctionality for the HarmoniQuA system:

� guideline tool: will generate guidelines from the KB;� monitoring tool: will monitor all activities withina modelling job and store these activities as a singlemodel journal in a model archive;

� report tool: generates reports from a model journal;� advisor tool: advises modellers in new modelling jobsbased on decisions and choices of previous jobs andassociated model journals in the model archive.

An overview of the HarmoniQuA products (KB andtools) and how these interact with the activities of theusers is presented in Fig. 4. The lower part of Fig. 4depicts the five major steps of the modelling process.These five major steps are decomposed into 45 tasks,with interrelations (order and feedback) as shown inFig. 5. Each task has an internal structure, i.e. name,definition, explanation, interrelations with other tasks,activities, activity related methods, references, taskinputs and outputs. This knowledge structure (steps,tasks, within-task-knowledge) is stored in the KB. Thefive steps and the tasks have been selected on the basis ofexisting modelling protocols and QA guidelines andinclude the key elements outlined in Section 3.3 above.

Model based decision support has several dimen-sions, which hinder a ‘one-size-fits-all’-approach. Har-moniQuA attempts to serve several types of users in

Model StudyPlan

Data andConceptualisation

ModelSet-up

Calibrationand

Validation

Simulation andEvaluation

Model Team

Single/multiple domain

Reporting and client review take place in each step

MoST

User

Knowledge Base

GuidelinesSoftware capabilitiesGlossaryDomains:

GroundwaterPrecipitation-runoffHydrodynamicsFlood forecastingWater qualityBiota (ecology)Socio-economics

Guidance

Generic + specific for:- model domain- user- job complexity

Reporting

Specific for typesof users

Monitoring

Generic + specific for:- model domain- user- job complexity

Advise

From previousmodel projects

Model

ArchiveModel journal, Project A

Model journal, Project B

Model journal, Project C

Model journal, Project D

Fig. 4. HarmoniQuA tools (MoST) to support the QA process.


Data an

d C

on

cep

tu

alisatio

nM

od

el S

tu

dy P

lan

Ord

in

ary task

Review

task

Decisio

n task

Le

ge

nd

s

Mo

del S

et-u

pC

alib

ratio

n a

nd

V

alid

atio

nS

im

ula

tio

n a

nd

E

va

lu

atio

n

Describ

e P

ro

blem

an

d

Co

ntext

Id

en

tify D

ata A

vailab

ility

Defin

e O

bjectives

Determ

in

e R

eq

uirem

en

ts

Prep

are T

erm

s o

f

Referen

ce

Pro

po

sal an

d T

en

derin

g

Pro

cess R

aw

D

ata

Describ

e S

ys

te

m a

nd

Data A

va

ila

bility

Su

ffic

ie

nt

Da

ta

?

Rep

ort an

d R

ev

is

it

Mo

del S

tu

dy

P

la

n

(C

on

cep

tu

alis

atio

n)

Co

ns

tru

ct M

od

el

Ass

es

s

So

un

dn

ess o

f

Co

nc

ep

t-

ua

lis

atio

n

Re

vie

w

Co

ncep

tu

alis

atio

n a

nd

Mo

del S

et-u

p P

la

n

Ne

ed

fo

r

Alte

rn

ativ

e

Co

nce

ptu

al

Mo

de

ls

?

Co

de S

ele

ctio

n

Te

st R

un

s

Co

mp

le

te

d

Re

po

rt a

nd

R

ev

is

it

Mo

de

l S

tu

dy

P

la

n (M

od

el

Se

t-u

p)

Re

vie

w M

od

el S

et-u

p

an

d C

alib

ra

tio

n a

nd

Va

lid

atio

n P

la

n

Sp

ec

ify

o

r U

pd

ate

Ca

lib

ra

tio

n +

V

alid

atio

n

Ta

rg

ets

a

nd

C

rite

ria

Pro

cess M

od

el S

tru

ctu

re

Da

ta

Valid

atio

n

As

se

ss

So

un

dn

es

s o

f

Calib

ratio

n

As

se

ss

So

un

dn

es

s o

f

Valid

atio

n

Un

ce

rta

in

ty A

nalysis o

f

Ca

lib

ratio

n an

d

Valid

atio

n

Do

cu

men

t M

od

el S

co

pe

Re

po

rt an

d R

evisit

Mo

del S

tu

dy P

lan

(C

alib

ra

tio

n +

V

alid

atio

n)

Re

vie

w C

alib

ratio

n an

d

Va

lid

atio

n an

d

Sim

ulatio

n P

lan

Su

mm

arise

C

on

ce

ptu

al

Mo

del an

d A

ss

um

ptio

ns

Sim

ulatio

ns

An

alyse an

d In

terp

ret

Resu

lts

Ch

ec

k

Sim

ulatio

ns

As

se

ss

So

un

dn

ess o

f

Sim

ulatio

n

Un

certain

ty A

nalysis o

f

Sim

ulatio

n

Rep

ortin

g o

f S

im

ulatio

n

(in

cl. U

ncertain

ty)

Review

o

f S

im

ulatio

n

Mo

del S

tu

dy C

lo

su

re

Se

le

ct C

alib

ratio

n

Param

eters

Mo

del S

tru

ctu

re

a

nd

Pro

ce

ss

es

Se

le

ct O

ptim

isatio

n

Me

th

od

De

fin

e S

to

p C

riteria

Param

eter

Op

tim

isatio

n

Sp

ecify S

tag

es in

Ca

lib

ratio

n S

trateg

y

All C

alib

ratio

n

Stag

es

Co

mp

leted

?A

gree o

n

Mo

del S

tu

dy

P

lan

an

d

Bu

dg

et

Mo

del P

ara

me

te

rs

no

no

ba

d

ye

s

ba

d

ye

s

ye

s

OK

OK

OK

ye

s

no

ye

s

ye

s

ye

s

no

no

t O

K

no

ye

s

no

no

ye

s

no

ye

s

no

ba

d

ba

d

no

no

no

no

ye

s

ba

dn

o

ba

d

ye

s

no

t O

K

feed

fo

rw

ard

feed

fb

ack

Fig.5.Thefivestepsand45tasksofmodellingprocess

intheHarm

oniQ

uA

knowledgebase.


a series of water management domains, in jobs ofdiverse complexity and diverse application purpose.In this way, users working on a specific job will only beconfronted with guidelines, instructions, decisions andactivities that are relevant to their role in a particularmodelling job.

The HarmoniQuA tools have been developed inProtege2000 following an ontological approach. Moredetails can be found in Kassahun et al. (2004). The toolsare available on http://www.harmoniqua.org/.

4.2. Key elements

Some of the key features to be implemented in thenew HarmoniQuA guidelines are:

4.2.1. Interactive guidelinesThe dialogue between the different players is crucial

to ensure that the output from the modelling process isunderstandable for stakeholders and beneficial for theclient. The importance of involvement of stakeholderand public opinions are emphasised by Pahl-Wostl(2002) and addressed in some Type 3 guidelines (e.g.BDMF, 2000; Pascual et al., 2003). In HarmoniQuA,each of the five major steps (Fig. 5) is thereforeconcluded with a dialogue task, in terms of eithercontract negotiation (first step) or reviews (last foursteps). A dialogue task encourages the assessment of thepresent step and provides the opportunity to redefine thecontent of the model study plan for the next step basedupon the results and findings of the present step. Thesedialogue steps provide flexibility to the modelling studyand ensure that the tasks that have yet to be performedcan be modified according to the achieved results andperceptions of modeller and client.

4.2.2. Transparency and reproducibilityTransparency and reproducibility are important,

especially for large studies involving use of complexmodels. This will be ensured through the MonitoringTool which enables modelling teams, consisting ofmodellers, managers and auditors, to be guided throughthe modelling process, to monitor all modelling activitiesand to oversee the status of each task to perform. Withan increasing tendency to reuse existing models orrebuild them with additional data, modified conceptualmodels (revised model structure and/or inclusion ofadditional processes) and improved calibration andvalidation tests, this functionality of the MonitoringTool becomes very important.

4.2.3. Accuracy criteriaEstablishment of accuracy criteria for a modelling

study is a very important, but difficult, issue. Modellersoften establish numerical accuracy criteria in order toclassify the goodness of a given model (e.g. Henriksen

et al., 2003; Scholten and Van der Tol, 1998). Theseattempts are very useful in making the performancemore transparent and quantitative, but do not providean objective means to decide what the optimal accuracycriteria really should be in a given case. According toRefsgaard and Henriksen (2004) no universal accuracycriteria can be established, i.e. it is generally not possiblefrom a natural scientific point of view to tell whena model performance is good enough. Such acceptancecriteria will vary from case to case depending on thesocio-economic context, i.e. what is at stake in thedecisions to be supported by the model predictions. Anappropriate question may be: how do we translate the‘soft’ socio-economic objectives to ‘hard-core’ modelperformance criteria? This is obviously a challenge thatcannot be solved by natural science alone, but needs tobe addressed in a much broader context includingaspects of economy, stakeholder interests and riskperception.

Performance statistics must comprise quantifiableand objective measures. However numerical measurescannot stand alone. Often expert opinions are necessarysupplements.

4.2.4. Uncertainty assessmentsQuality assurance and uncertainty assessments are

two aspects that are very closely linked. Initially, themanager has to define accuracy criteria from a percep-tion of which uncertainty level he/she believes is suitablefor a particular case (see above). Subsequently, as themodelling study proceeds, the dialogue between mod-eller and manager has to continue with the necessarytrade off between modelling accuracy and the cost of themodelling study. In the uncertainty assessments it is veryimportant to go beyond the traditional statisticaluncertainty analysis. Thus, e.g. aspects of scenariouncertainty and ignorance should generally be includedand in addition the uncertainties originating from dataand models often needs to be integrated with socio-economic aspects in order to form a suitable basis forthe further decision process (e.g. Van Asselt andRotmans, 2002). Thus, like with the accuracy criteria(above) the use of uncertainty assessments in waterresources management goes beyond natural science.

Assessment of uncertainty due to errors in the modelstructure is a particularly difficult task and is most oftenneglected. One way of evaluating this source of un-certainty is through the establishment of alternativeconceptual models. This aspect is emphasised in theHarmoniQuA guidelines.

4.2.5. Model validationAlthough experience shows that models generally

perform poorer in validation tests against independentdata than they do in calibration tests, model validation isin our opinion a neglected issue, both in many modelling

http://www.harmoniqua.org/


guidelines and in the scientific literature. Maybe manyscientists have not wanted to use the term validation dueto the scientific philosophically related controversies, butin any casemany scientists are not advocating the need formodel validation. One of the unfortunate consequencesof this ‘lack of interest’ is that not much work has beendevoted to developing suitable validation test schemessince Klemes (1986). In our opinion further developmentof suitable testing schemes, particularly for non-linearmodels and for applications comprising extrapolationsbeyond the calibration data basis, and imposing them toall modelling projects is a major future challenge.

4.2.6. Dedication aspectsThe QA guidelines describe the different tasks and

responsibilities of the different types of users such as (1)modellers; (2) water managers; (3) auditors; (4) stake-holders (other than water manager); and (5) generalpublic.

The QA guidelines are developed so that theyadequately reflect the different requirements in severalmodelling domains (and still maintain a common genericcore to ensure coherency). Furthermore, the guidelineswill be applicable for studies where several domains,including socio-economy, are integrated.

The QA guidelines differentiate according to jobcomplexity in modelling, e.g. (1) basic (rough calcu-lations); (2) intermediate (moderately complex calcula-tions); and (3) comprehensive (sophisticated, detailedcalculations).

5. Discussion and conclusions

5.1. Types and reasons of existing QA guidelines

Wehave classified quality assurance (QA) guidelines inthree types: Internal technical guidelines (Type 1), Publictechnical guidelines (Type 2), and Public interactiveguidelines (Type 3). We have then characterised theconditions for which the guidelines are used by (a) thescientific maturity of the underlying discipline(s) and (b)the maturity of the modelling market in the region/country for which the guidelines were developed. Ourreview of existing QA guidelines is not exhaustive, butlimited to examples aimed at being representative forconditions in Europe, North America and Australia.Thus, we have for instance not reviewed QA guidelinesfrom countries in Asia, where modelling has taken placefor many years. The results of our review revealedsignificant variations in the type of guidelines availableand their usage between different modelling domains andcountries.We hypothesised that the stage ofQA guidelinedevelopment largely depends on the maturity of both thespecific scientific discipline and the modelling market inthe respective country or region (Figs. 2 and 3).

Considering Figs. 2 and 3 it appears that the maturityof the scientific discipline and market both play animportant role in QA development. However, neitherthe scientific level nor the market maturity alone is ableto explain the differences in the stage of QA guidelinedevelopment. If the underlying process understanding ornecessary data are too weak, then the modelling processlacks credibility no matter how well QA procedures areadhered to. Hence, the motivation to establish sophis-ticated QA guidelines in such cases is small. Similarly,even though a specific discipline may be scientificallymature, modellers may be reluctant to use sophisticatedQA guidelines if they are not required to do so byregulators and/or water managers. The general de-velopment of QA guidelines has progressed over timefrom Type 1 towards Type 3. A developmental processthat is consistent with the results of the reviews asreflected in Figs. 2 and 3 is the following.

Initially, when models are introduced for practicalapplication, internal technical guidelines (Type 1)originating from the research community are applied.The development from Type 1 to Type 2 QA guidelinesrequires a certain degree of maturity within both thespecific scientific discipline and the market. This impliesthat there should not be significant lacks of knowledgeon process descriptions, and that there is a commonagreement about the scientifically sound procedures forsolving the problems in this domain. The developmentof Type 2 guidelines is most often driven by the demandsof regulators and water managers. The developmentfrom Type 2 to Type 3 requires a clear and consciousdemand from regulators and water managers.

It would also have been possible to classify the QAguidelines after other criteria, for example according tohow uncertainty analysis is treated, whether they applyto single or multiple domains and whether they apply tonatural or social science. We have chosen our classifica-tion for two main reasons. Firstly, an improved mutualunderstanding between modeller and water manager iscrucial for a model application to be successful inpractice, and this should be facilitated by the QAguidelines. Secondly, the trend of increasing stakeholderinvolvement in the water resources management processdemands that QA guidelines also enable stakeholders toobserve and take part in parts of the modelling process.

Our characterisation of QA guidelines according toscientific and market maturity has some weaknesses.First of all, the assessments have been done subjectively,because there was no other feasible method. Secondly,the two characteristics are not completely independent.Thus a large and mature market will often put demandson new scientific knowledge and hence to enhance thescientific development, as well as it will lead to needs forimproved technical standards.

Altogether, it may be concluded that our hypotheseson the importance of scientific and market maturity for


the development of QA guidelines have not beenfalsified. However, due to the above weaknesses andthe limited empirical basis (review not exhaustive butselected examples) this conclusion should be taken withsome reservation.

5.2. Organisational requirementsfor QA guidelines to be effective

As emphasised by e.g. Forkel (1996) modellingstudies involve several partners with different responsi-bilities. The ‘key players’ are code developers, modelusers (modellers) and water managers (including plan-ning and regulatory authorities). To a large extent thequality of the modelling study is determined by theexpertise, attitudes and motivation of the teams involvedin the modelling and quality assessment process.

The attitude of the modellers is important. NRC(1990) characterises this as follows: ‘‘most modellersenjoy the modelling process but find less satisfaction inthe process of documentation and quality assurance’’.Scholten and Groot (2002) describe the main problemwith the Dutch Handbook on Good Modelling Practicethat they all like it, but only a few use it.

QA will only become successful if both of the parties,modeller and water manager, are motivated and active insupporting its use. The water manager has a particularresponsibility, because he/she has the power to requestand pay for adequate QA inmodelling studies. Therefore,QA guidelines can only be expected to be used in practice,if the water manager prescribes their use. In this respect itis very important that thewatermanager has the technicalcapacity to organise the QA process. A significantproblem for water manager’s organisation is that it oftenlacks individuals who are trained at an appropriate levelto understand and use models. If the water manager doesnot possess such skill within his/her own staff, an externalmodelling expert can be hired to help the manager in theQA process. However, this requires that the manager isaware of the problem and the need.

5.3. The HarmoniQuA guidelines

The approach adopted in the present HarmoniQuAguidelines correspond to Type 3. However, in additionto its focus on the dialogue and role play between thevarious actors in the modelling process, i.e. modellers,water managers, auditors and the public/stakeholders,the HarmoniQuA approach is innovative compared toexisting Type 3 QA guidelines on the following aspects:

� Supporting software tools, beyond simple score-boards and templates, are novel and importantelements. These tools, which contain the knowledgebase (KB), can guide the users through themodelling process, monitor decisions and outcomes,

and provide experienced based advise on theappropriate route to be followed. This will signifi-cantly improve the transparency and reproducibilityof the modelling process. To our knowledge no suchtools exist or are under development at present.

� The focus on performance and accuracy criteriain the modelling process is not novel as such. How-ever, the current adaptation of these criteria throughthe process in connection with the formalised reviewsteps is, if not novel, then at least emphasised muchmore in the HarmoniQuA guidelines than in anyother existing guidelines. This approach allows theHarmoniQuA guidelines to fit nicely with the newideas of adaptive management (Pahl-Wostl, 2002).

� The uncertainty aspects are given a more central rolethan in existing guidelines, where uncertainty oftenis confined to assessment of predictive uncertaintiestowards the end of the study. In the HarmoniQuAguidelines uncertainty aspects plays an importantrole in 13 of the 45 tasks. Thus, uncertaintyassessment is a central element in the dialoguebetween modeller and water manager already in thebeginning of the model study when the initialperformance criteria are outlined. Furthermore,HarmoniQuA recommends including less quantifi-able elements such as scenario uncertainty andmodel structural uncertainty in the assessment.

� Model validation tests against independent data havemore emphasis than in most other guidelines.Although the most comprehensive of the existingguidelines, the Dutch guidelines (Van Waverenet al., 2000), for example recommends validationto be carried out, they do not describe validationtests beyond the traditional split-sample test.

� The HarmoniQuA guidelines are unique in theirdedication aspects, namely that different tasks andresponsibilities are described for different users,different modelling domains and different levels ofmodelling job complexity. The Australian ground-water modelling guidelines have the same feature,but only with respect to the review procedures(Merrick et al., 2002).

The HarmoniQuA guidelines consist of a comprehen-sive set of QA guidelines for multiple modelling domainscombined with the supporting software tools. Thesefunctionalities appear to be well suited to the challengesand demands of modern water resources management.The usefulness, user friendliness and appreciation by theusers will be assessed through a testing of the guidelinesand tools in a range of river basin modelling projects.

Acknowledgements

The present work was carried out within theProject ‘Harmonising Quality Assurance in model based


catchments and river basin management (Harmoni-QuA)’, which is partly funded by the EC Energy,Environment and Sustainable Development programme(Contract EVK1-CT2001-00097). The constructive com-ments of five anonymous reviewers are acknowledged.

References

Anderson, M.G., Bates, P.D., 2001. Hydrological science: model

credibility and scientific integrity. In: Anderson, M.G., Bates, P.D.

(Eds.), Model Validation. Perspectives in Hydrological Science.

John Wiley & Sons, Chichester, pp. 1–10.

Anderson, M.P., Woessner, W.W., 1992. The role of postaudit in

model validation. Advances in Water Resources 15, 167–173.

Anderson, M.P., Ward, D.S., Lappala, E.G., Prickett, T.A., 1993.

Computer models for subsurface water. In: Maidment, D.R. (Ed.),

Handbook of Hydrology. McGraw-Hill, Inc (Chapter 22).

ASTM, 1992. Standard Practice for Evaluating Mathematical Models

for the Environmental Fate of Chemicals. Standard E978-92,

American Society for Testing and Materials, http://www.astm.org.

ASTM, 1994. Standard Guide for Application of a Ground-Water

Flow Model to a Site-Specific Problem. Standard D5447-93,

American Society for Testing and Materials, http://www.astm.org.

Balint, G., 2002. State-of-the-art for flood forecasting modelling. In:

Refsgaard, J.C. (Ed.), State-of-the-Art Report on Quality Assur-

ance in Modelling Related to River Basin Management. Chapter 7,

Geological Survey of Denmark and Greenland, Copenhagen,

http://www.harmoniqua.org.

BDMF, 2000. Protocols for Water and Environmental Modeling.

Bay-Delta Modeling Forum. Ad hoc Modeling Protocols Com-

mittee, http://www.sfei.org/modelingforum/.

Blind, M., 2004. ICT requirements for an ‘evolutionary’ development

of WFD compliant River Basin Management Plans. In: Pahl, C.,

Schmidt, S., Jakeman, T. (Eds.), iEMSs 2004 International

Congress: ‘‘Complexity and Integrated Resources Management’’.

International Environmental Modelling and Software Society,

Osnabruck, Germany, June 2004.

CAMASE, 1996. CAMASE was a Concerted Action for the Develop-

ment and Testing of Quantitative Methods for research on

Agricultural Systems and the Environment, http://www.bib.wau.

nl/camase/.

Da Silva, M.C., Barbosa, A.E., Rocha, J.S., Fortunato, A.B., 2002.

State-of-the-art for surface water quality modelling. In: Refsgaard,

J.C. (Ed.), State-of-the-Art Report on Quality Assurance in

Modelling Related to River Basin Management. Chapter 8,



DHI, 2002. MIKE 11 User Guide. DHI Water & Environment,

Hørsholm, Denmark.

Forkel, C., 1996. Das numerische Modell – ein schmaler Grat zwischen

vertrauenswurdigem Werkzeug und gefahrlichem Spielzeug. Pre-

sented at the 26. IWASA, RWTH Aachen, 4–5 January 1996.

GWP-TAC, 2000. Integrated Water Management, TEC Background

Papers No. 4, Global Water Partnership, SE-105 25 Stockholm,

Sweden, ISBN: 91-630-9229-8.

Heinz, I., Eberle, S., 2002. State-of-the-art for socio-economic

modelling. In: Refsgaard, J.C. (Ed.), State-of-the-Art Report on

Quality Assurance in Modelling Related to River Basin Manage-

ment. Chapter 10, Geological Survey of Denmark and Greenland,

Copenhagen, http://www.harmoniqua.org.

Henriksen, H.J., 2002a. Australian groundwater modelling guidelines.

In: Refsgaard, J.C. (Ed.), State-of-the-Art Report on Quality

Assurance in Modelling Related to River Basin Management.

Chapter 13, Geological Survey of Denmark and Greenland,

Copenhagen, http://www.harmoniqua.org.

Henriksen, H.J., 2002b. Danish groundwater modelling guidelines. In:


ance in Modelling Related to River Basin Management. Chapter

14, Geological Survey of Denmark and Greenland, Copenhagen,


Henriksen, H.J., Troldborg, L., Nyegaard, P., Sonnenborg, T.O.,

Refsgaard, J.C., Madsen, B., 2003. Methodology for construction,

calibration and validation of a national hydrological model for

Denmark. Journal of Hydrology 280 (1–4), 52–71.

Kassahun, A., Scholten, H., Zompanakis, G., Gavardinas, C., 2004.

Support for model based water management with the Harmo-

niQuA toolbox. In: Pahl, C., Schmidt, S., Jakeman, T. (Eds.),

iEMSs 2004 International Congress: ‘‘Complexity and Integrated

Resources Management’’. International Environmental Modelling

and Software Society, Osnabruck, Germany, June 2004.

Klemes, V., 1986. Operational testing of hydrological simulation

models. Hydrological Sciences Journal 31, 13–24.

Merrick, N.P., Middlemis, H., Ross, J.B., 2002. Groundwater

Modelling Guidelines for Australia – Recommended Procedures

for Modelling Reviews. International Groundwater Conference.

Balancing the Groundwater Budget. Northern Territory. Australia.

12–17 May 2002.

Metelka, T., Krejcik, J., 2002a. State-of-the-art for hydrodynamic. In:


ance in Modelling Related to River Basin Management. Chapter 6,



Metelka, T., Krejcik, J., 2002b. Quality assurance in Central and

Eastern Europe. In: Refsgaard, J.C. (Ed.), State-of-the-Art Report

on Quality Assurance in Modelling Related to River Basin

Management. Chapter 15, Geological Survey of Denmark and

Greenland, Copenhagen, http://www.harmoniqua.org.

Middlemis, H., 2000. Murray-Darling Basin Commission. Ground-

water Flow Modelling Guideline. Aquaterra Consulting Pty Ltd.

South Perth. Western Australia. Project no. 125.

NRC, 1990. Ground Water Models: Scientific and Regulatory

Applications. National Research Council, National Academy

Press, Washington, D.C.

Old, G.H., Packman, J.C., Calver, A.N., 2002. State-of-the-art

for biota (ecological) modelling. In: Refsgaard, J.C. (Ed.),

State-of-the-Art Report on Quality Assurance in Modelling

Related to River Basin Management. Chapter 9, Geological

Survey of Denmark and Greenland, Copenhagen, http://www.

harmoniqua.org.

Packman, J.C., 2002. Quality Assurance in the UK. In: Refsgaard, J.C.

(Ed.), State-of-the-Art Report on Quality Assurance in Modelling



harmoniqua.org.

Pahl-Wostl, C., 2002. Towards sustainability in the water sector – the

importance of human actors and processes of social learning.

Aquatic Sciences 64, 394–411.

Pascual, P., Stiber, N., Sunderland, E., 2003. Draft Guidance on the

Development, Evaluation, and Application of Regulatory Envi-

ronmental Models. Council for Regulatory Environmental Mod-

eling. US EPA, Washington D.C.

Perrin, C., Andreassian, V., Michel, C., 2002a. State-of-the-art for

precipitation-runoff modelling. In: Refsgaard, J.C. (Ed.), State-

of-the-Art Report on Quality Assurance in Modelling Related to

River Basin Management. Chapter 5, Geological Survey of Den-

mark and Greenland, Copenhagen, http://www.harmoniqua.org.

Perrin, C., Andreassian, V., Michel, C., 2002b. Quality assurance for

precipitation-runoff modelling in France. In: Refsgaard, J.C. (Ed.),

State-of-the-Art Report on Quality Assurance in Modelling



harmoniqua.org.

http://www.astm.org

http://www.astm.org


http://www.sfei.org/modelingforum/

http://www.bib.wau.nl/camase/

http://www.bib.wau.nl/camase/















Refsgaard, J.C. (Ed.), 2002. State-of-the-Art Report on Quality

Assurance in Modelling Related to River Basin Management.

Report from the EU research project HarmoniQuA, http://www.

harmoniqua.org. 18 chapters, 182 pp. Geological Survey of

Denmark and Greenland, Copenhagen.

Refsgaard, J.C., Henriksen, H.J., 2002. State-of-the-art for Ground-

water Modelling. In: Refsgaard, J.C. (Ed.), State-of-the-Art Report

on Quality Assurance in Modelling Related to River Basin

Management. Chapter 4, Geological Survey of Denmark and

Greenland, Copenhagen, http://www.harmoniqua.org.

Refsgaard, J.C., Henriksen, H.J., 2004. Modelling guidelines –

terminology and guiding principles. Advances in Water Resources

27, 71–82.

Rumbaugh, J.O., Rumbaugh, D.B., 2001. Guide to Using Ground-

water Vistas. Environmental Simulations, Inc, Virginia, USA.

Rykiel, E.R., 1996. Testing ecological models: the meaning of

validation. Ecological Modelling 90, 229–244.

Scholten, H., Van der Tol, M.W.M., 1998. Quantitative validation of

deterministicmodels: when is amodel acceptable? In: Obaidat,M.S.,

Davoli, F., DeMarinis, D. (Eds.), The Proceedings of the Summer

Computer Simulation Conference. SCS, The Society for Computer

Simulation International, San Diego, CA, USA, pp. 404–409.

Scholten, H., Groot, S., 2002. Dutch guidelines. In: Refsgaard, J.C.

(Ed.), State-of-the-Art Report on Quality Assurance in model-

ling related to river basin management. Chapter 12, Geological


harmoniqua.org.

Scholten, H., Van Waveren, R.H., Groot, S., Van Geer, F.C., Wosten,

J.H.M., Koeze, R.D., Noort, J.J., 2000. Good Modelling Practice

in Water Management. Paper Presented on Hydroinformatics

2000, Cedar Rapids, IA, USA.

Van Asselt, M.B.A., Rotmans, J., 2002. Uncertainty in integrated

assessment modelling – From positivism to pluralism. Climatic

Change 54 (1–2), 75–105.

Van Gils, J.A.G., Groot, S., 2002. Examples of good modelling

practice in the Danube Basin. In: Refsgaard, J.C. (Ed.), State-

of-the-Art Report on Quality Assurance in Modelling Related to

River Basin Management. Chapter 18, Geological Survey of

Denmark and Greenland, Copenhagen, http://www.harmoniqua.

org.

Van Waveren, R.H., Groot, S., Scholten, H., Van Geer, F.C., Wosten,

J.H.M., Koeze, R.D., Noort, J.J., 2000. Good Modelling Practice

Handbook, STOWA Report 99-05, Utrecht, RWS-RIZA, Lelys-

tad, The Netherlands, http://waterland.net/riza/aquest/ (In Dutch).








http://waterland.net/riza/aquest/

Documents

Quality assurance in model based water management - review of existing practice and outline of new approaches