15
Atmospheric Environment 40 (2006) 4920–4934 An objective comparison of CMAQ and REMSAD performances Edith Ge´go a, , P. Steven Porter b , Christian Hogrefe c , John S. Irwin d a 308 Evergreen Drive, Idaho Falls, ID 83401, USA b University of Idaho, 1776 Science Center Drive, Idaho Falls, ID 83402, USA c Atmospheric Sciences Research Center, University at Albany, ES 351, State University of New York, Albany, NY 12222, USA d 1900 Pony Run Road, Raleigh, NC 27615-7415, USA Received 8 February 2005; received in revised form 17 October 2005; accepted 22 December 2005 Abstract Photochemical air quality modeling systems are the primary tools used in regulatory applications to assess the impact of different emission reduction strategies aimed at reducing air pollutant concentrations to levels considered safe for public health. Two such modeling systems are the community multiscale air quality (CMAQ) model and the regional modeling system for aerosols and deposition (REMSAD). To facilitate their inter-comparison, the United States Environmental Protection Agency performed simulations of air quality over the contiguous United States during year 2001 (horizontal grid cell size of 36 36 km) with CMAQ and REMSAD driven by identical emission and meteorological fields. Here, we compare the abilities of CMAQ and REMSAD to reproduce measured aerosol nitrate and sulfate concentrations. Model estimates are compared to observations reported by the interagency monitoring of protected visual environment (IMPROVE) and the clean air status and trend network (CASTNet). Root mean squared errors are calculated for simulation/observation pairs from ten geographic regions and 12 seasons (months). Following the application of the Wilcoxon matched-pair signed rank test, we conclude that CMAQ is more skillful than REMSAD for simulation of aerosol sulfate. Simulations of particulate nitrate concentrations by CMAQ and REMSAD can seldom be differentiated, leading to the conclusion that both models perform equally for this pollutant specie. r 2006 Elsevier Ltd. All rights reserved. Keywords: Aerosol sulfate; Aerosol nitrate; Wilcoxon signed rank test; Evaluation metric; Photochemical model 1. Introduction Models are the principal tools used by govern- mental agencies to develop emission reduction strategies aimed at achieving safe and therefore admissible air quality. Models are indeed the only tool that allows testing of the impact of different reduction strategies on air quality and, therefore, facilitate decisions about the most suitable alter- natives. Two of the most prominent modeling systems are the community multiscale air quality (CMAQ) ARTICLE IN PRESS www.elsevier.com/locate/atmosenv 1352-2310/$ - see front matter r 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.atmosenv.2005.12.045 Corresponding author. Tel.: +1 208 523 5873; fax: +1 208 282 7975. E-mail address: [email protected] (E. Ge´go).

An objective comparison of CMAQ and REMSAD performances

Embed Size (px)

Citation preview

Page 1: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESS

1352-2310/$ - se

doi:10.1016/j.at

�Correspondfax: +1208 282

E-mail addr

Atmospheric Environment 40 (2006) 4920–4934

www.elsevier.com/locate/atmosenv

An objective comparison of CMAQ andREMSAD performances

Edith Gegoa,�, P. Steven Porterb, Christian Hogrefec, John S. Irwind

a308 Evergreen Drive, Idaho Falls, ID 83401, USAbUniversity of Idaho, 1776 Science Center Drive, Idaho Falls, ID 83402, USA

cAtmospheric Sciences Research Center, University at Albany, ES 351, State University of New York, Albany,

NY 12222, USAd1900 Pony Run Road, Raleigh, NC 27615-7415, USA

Received 8 February 2005; received in revised form 17 October 2005; accepted 22 December 2005

Abstract

Photochemical air quality modeling systems are the primary tools used in regulatory applications to assess the impact of

different emission reduction strategies aimed at reducing air pollutant concentrations to levels considered safe for public

health. Two such modeling systems are the community multiscale air quality (CMAQ) model and the regional modeling

system for aerosols and deposition (REMSAD). To facilitate their inter-comparison, the United States Environmental

Protection Agency performed simulations of air quality over the contiguous United States during year 2001 (horizontal

grid cell size of 36� 36 km) with CMAQ and REMSAD driven by identical emission and meteorological fields. Here, we

compare the abilities of CMAQ and REMSAD to reproduce measured aerosol nitrate and sulfate concentrations. Model

estimates are compared to observations reported by the interagency monitoring of protected visual environment

(IMPROVE) and the clean air status and trend network (CASTNet). Root mean squared errors are calculated for

simulation/observation pairs from ten geographic regions and 12 seasons (months). Following the application of the

Wilcoxon matched-pair signed rank test, we conclude that CMAQ is more skillful than REMSAD for simulation of

aerosol sulfate. Simulations of particulate nitrate concentrations by CMAQ and REMSAD can seldom be differentiated,

leading to the conclusion that both models perform equally for this pollutant specie.

r 2006 Elsevier Ltd. All rights reserved.

Keywords: Aerosol sulfate; Aerosol nitrate; Wilcoxon signed rank test; Evaluation metric; Photochemical model

1. Introduction

Models are the principal tools used by govern-mental agencies to develop emission reduction

e front matter r 2006 Elsevier Ltd. All rights reserved

mosenv.2005.12.045

ing author. Tel.: +1 208 523 5873;

7975.

ess: [email protected] (E. Gego).

strategies aimed at achieving safe and thereforeadmissible air quality. Models are indeed the onlytool that allows testing of the impact of differentreduction strategies on air quality and, therefore,facilitate decisions about the most suitable alter-natives.

Two of the most prominent modeling systems arethe community multiscale air quality (CMAQ)

.

Page 2: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–4934 4921

model (Byun and Ching, 1999) and the regionalmodeling system for aerosols and deposition (RE-MSAD) (ICF Consulting, 2002). To promotemodel-to-model comparison of these two modelingsystems, the United States Environmental Protec-tion Agency (US EPA) recently used CMAQ andREMSAD to simulate air quality over the contig-uous US during year 2001, with both modelsresponding to identical inputs (meteorology, emis-sions, etc.). Our objective is to use the results ofthese simulations to compare the ability of RE-MSAD and CMAQ to reproduce measured aerosolnitrate and sulfate concentrations.

Our evaluation of the respective strengths andweaknesses of CMAQ and REMSAD relies oncalculation of the root mean squared errors(RMSE) between model estimates and correspond-ing observations. In an effort to unveil the areas andtime periods where the quality of CMAQ andREMSAD estimates significantly differ from eachother, simulation results were organized into tengeographical areas and monthly periods for calcula-tion of the evaluation metric (RMSE). Objectivity inour assessment is attained by submitting matchingsets of the evaluation metric characterizing CMAQand REMSAD respectively, to a statistical test ofcomparison of means.

2. Models

Air quality estimates were produced by CMAQ(2004 release version) and REMSAD (version 7.6)using nearly identical meteorological and emission

CASTNet observation site

IMPROVE observation site

Region III

Region IV

Region V

Reg

Region I

Region II

Fig. 1. Regions identified in the contiguous US and loca

fields. CMAQ and REMSAD are three-dimensionalEulerian air quality modeling systems designedto simulate the chemistry, transport, and depositionof airborne pollutants. The two systems mostlydiffer from each other by their modeling ofchemistry. Details about CMAQ and REMSADcan be found at http://www.epa.gov/asmdnerl/models3/doc/science/ and http://remsad.saintl.com,respectively.

The meteorological fields used in CMAQ andREMSAD were produced by MM5, the fifthgeneration Penn State University (PSU)/ NationalCenter for Atmospheric Research (NCAR) mesos-cale model (Grell et al., 1994). MM5 (version 5) wasused to reconstruct meteorology over the continen-tal United States from 1 January 2001 to 31December 2001 with a horizontal resolution of36 km. Vertically, the domain comprises 34 layerswith the surface layer approximately 50m deep.Topographic information was developed using theNCAR and the United States Geological Survey(USGS) terrain databases. Vegetation type and landuse information was developed using the NCAR/PSU databases provided with MM5. Initial andboundary conditions were extracted from theNCAR ETA reanalysis archives. An analysis-nud-ging technique was used to nudge predictions(winds, temperature and the mixing ratio) towardsurface and aloft observations. Thermodynamicvariables were not nudged within the boundarylayer. The model was run with a 51/2 day windowand a restart at 12:00 GMT (Greenwich mean time)every fifth day. Further details about the MM5

Region IX

Region VIII

Region VI

ion VII Region X

tion of the observation sites included in the study.

Page 3: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–49344922

setting, such as the physical options utilized, areavailable in McNally (2003).

The MM5 fields were processed by the meteorol-ogy-chemistry interface preprocessor (MCIP) ver-sion 2.2 to provide linkage to the air quality models.See details about MCIP at http://www.epa.gov/asmdnerl/models3/doc/science/chap12.pdf.

Anthropogenic emission fields from fixed sourceswere obtained with the sparse matrix operatorkernel emission model (SMOKE) (Carolina Envir-onmental Programs, 2003) processing the US EPANational Emissions Inventory for 2001. Emissions

Table 1

Number of IMPROVE and CASTNet sites in each region

Region Total

I II III IV V VI VII VIII IX X

CASTNet 3 7 3 8 1 14 12 6 14 5 73

IMPROVE 9 12 12 24 5 4 6 4 5 5 86

Table 2

Comparison of CMAQ and REMSAD estimates of sulfate concentrati

Month Region

I II III IV V

RMSE (mg m�3) characterizing CMAQ estimates, by month and region

Jan. 0.71 0.35 0.38 0.61 0.76Feb. 0.38 0.29 0.41 0.34 0.88Mar. 0.30 0.76 0.31 0.49 0.68Apr. 0.47 0.54 0.35 0.44 0.72May 0.40 0.96 0.34 0.43 0.61Jun. 0.51 1.08 0.26 0.56 0.59Jul. 0.87 1.26 0.20 0.61 0.65Aug. 0.72 1.21 0.25 0.55 0.52Sep. 0.47 0.99 0.25 0.45 0.48Oct. 0.27 1.11 0.23 0.39 0.43Nov. 0.30 0.52 0.26 0.84 0.54Dec. 0.24 0.34 0.24 0.32 0.55

RMSE (mg m�3) characterizing REMSAD estimates, by month and reg

Jan. 0.74 0.34 0.38 0.65 0.95Feb. 0.36 0.28 0.35 0.38 0.93Mar. 0.35 0.75 0.33 0.43 0.73Apr. 0.38 0.57 0.41 0.58 0.90May 0.41 0.86 0.40 0.44 0.47Jun. 0.52 1.00 0.27 0.63 0.49Jul. 0.91 1.21 0.23 0.70 0.68Aug. 0.79 1.13 0.31 0.62 0.61Sep. 0.43 0.94 0.26 0.57 0.47Oct. 0.28 0.99 0.22 0.54 0.39Nov. 0.32 0.49 0.29 0.96 0.41Dec. 0.24 0.35 0.19 0.18 0.59

from mobile sources were prepared with theMOBILE 6 module (US EPA, 2003); biogenicemissions were estimated with BEIS3.12 (http://www.epa.gov/asmdnerl/biogen.html) in conjunctionwith the MM5-derived meteorological estimates.Model-ready emission data with a horizontal gridsize of 36 km� 36 km were created from theemission fields by the emission-chemistry interfaceprocessor (ECIP).

3. Observations

Observations used to judge model performanceare aerosol sulfate and nitrate concentrationsreported by the interagency monitoring of protectedvisual environment (IMPROVE) network and theclean air status and trend network (CASTNet). TheIMPROVE network was designed to supervise airquality in pristine environments whereas CASTNetsites are located mostly in rural, not necessarilypristine, situations. In the western United States,though, newly added CASTNet sites are often

on to observations at IMPROVE sites

VI VII VIII IX X

0.59 1.39 0.83 1.31 1.31

0.48 3.83 0.79 0.97 1.19

0.71 0.98 0.62 1.34 1.65

1.10 1.30 0.72 1.24 1.36

1.15 1.87 1.47 3.09 2.11

1.44 2.17 1.51 2.65 2.24

1.63 2.61 1.57 1.89 2.09

1.16 2.91 1.78 9.59 2.22

0.92 2.24 1.71 2.61 1.78

0.62 2.13 1.32 1.91 1.50

1.10 1.45 1.25 1.03 1.12

0.64 0.92 1.10 2.13 1.31

ion

0.64 1.59 1.07 1.30 1.15

0.50 3.80 0.65 1.05 0.96

0.89 0.97 0.78 1.31 1.56

1.00 2.41 1.49 1.61 1.42

0.52 2.26 1.87 3.48 2.40

1.38 2.95 2.56 3.34 1.86

1.75 4.66 3.55 2.21 2.58

1.23 3.92 2.25 11.00 2.86

1.35 2.24 1.23 2.88 1.35

0.42 0.75 1.55 0.83 0.76

0.84 0.95 1.09 1.09 1.17

0.64 0.79 1.04 3.02 1.03

Page 4: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–4934 4923

collocated with an IMPROVE counterpart. Sam-pling and analysis protocols adopted in CASTNetand IMPROVE are also very different: CASTNetsamples are 7-day integrated averages while IM-PROVE samples are 24-h averages collected everythird day.

The air sampler at a CASTNet site is a non-sizeselective three-stage filter pack located 10m aboveground level. Filters are not equipped with a particlesize-limiting device but the flow rate utilized and theheight of the instrument are thought to prohibitentrance of coarse particles (Finkelstein, 2003,personal communication). The nitrate and sulfateions interpreted as particulate species are collectedon the first of the three consecutive filters, composedof Teflon. CASTNet concentrations are standar-dized to a temperature of 25 1C and a pressure of1013mb before being reported.

The IMPROVE air sampler consists of fourmodules located 3m above ground level andequipped with a device that stops particles largerthan 2.6 mm. Sulfate concentration is determinedfrom the sulfur found on a teflon filter. Nitrate isdetermined from particles caught on a nylon filterthat is preceded by an acidic vapor diffusion

Table 3

Differentiation of the performances of CMAQ and REMSAD to sim

T 0—probability levels p less than 5% are underlined)

Season Network Region

I II III

Within each region

All year IMPROVE T 0 29 8 26

p (%) 23.49 0:61R

16.9

CASTNet T 0 32 6 19

p (%) 31.10 0:34C

6.47

High conc. months IMPROVE T 0 10 1 0

p (%) 50.0 3:13R

1:56C

CASTNet T 0 5 0 0

p (%) 15.63 1:56C

1:56C

Network Month

Jan. Feb. Mar. Apr. May.

Within each month

IMPROVE T 0 12 22 21 9 19

p (%) 6.54 31.25 27.83 3:22C

21.58

CASTNet T 0 15 21 16 4 0

p (%) 11.62 27.83 13.77 0:68C

0:10C

C: indicates that CMAQ is significantly better than REMSAD; R: ind

denuder which eliminates nitric acid vapor (non-particulate nitrate). Measured concentrations arereported at ambient temperature and pressureconditions.

While aerosol sulfates are collected on the sametype of substrate (Teflon filter) at both IMPROVEand CASTNET sites, nitrate interpreted as particu-late material is collected on a Teflon filter atCASTNet sites and a nylon filter at IMPROVEsites. These differences in sampling equipmentprobably justify differences between nitrate concen-trations reported by the two networks at almostcollocated sites (Gego et al, 2005). Further detailsabout the sampling protocols utilized by eachnetwork, as well as the data they provide areavailable at http://vista.cira.colostate.edu/IMPROVE/and http://www.epa.gov/CASTNet.

4. Methods

4.1. Model evaluation metric

A variety of evaluation metrics are available toassess a model’s ability to reproduce past observa-tions (e.g., Hanna, 1994). For the model-to-model

ulate aerosol sulfate with Wilcoxon Matched Pair test (statistic

IV V VI VII VIII IX X

14 31 36 28 15 14 31

7 2:61C

28.47 42.50 21.19 3:20C

2:6C

28.47

0 6 20 17 6 21 270:02C

0.34 7.57 4:6C

0:34C

8.81 19.02

0 10 10 1 3 0 71:56C

50 50. 3:13C

7.81 1:56C

450

0 5 0 0 0 0 11:56C

15.63 1:56C

1:56C

1:56C

1:56C

3:13C

Jun. Jul. Aug. Sep. Oct. Nov. Dec.

21 4 6 23 14 18 17

27.83 0:69C

1:37C

34.77 9.67 18.75 16.11

0 2 0 8 19 7 240:10C

0:29C

0:10C

2:44C

21.58 1:86C

38.48

icates that REMSAD is significantly better than CMAQ.

Page 5: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–49344924

comparison presented here, we chose to utilize theRMSE. This metric seemed better suited for ourpurpose than the model bias, other very commonlyused evaluation metric, because it is calculated asthe average of positive values only (the squarederrors) and, therefore, will not dissimulate potentialmodel flaws that occurs when positive and negativeerrors are added. The RMSE between a set of modelpredictions and the observed values the model issupposed to reproduce is calculated as

RMSE ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1

n

XðMðx; tÞ �Oðx; tÞÞ2

r.

With M(x,t): model prediction at location x andtime t, O(x,t): observation at location x and time t,n: number of model/observation pairs.

Because of the sampling protocol differencesmentioned above, RMSEs are calculated separatelyfor the two networks. Model grid cell estimates arecompared to data collected at the observation sitelocated in the corresponding cell. No interpolationwas carried out to account for the change of supportvolume from a model cell average to a pointmeasurement.

Time (day)

Time (day)

Con

cent

ratio

n (µ

g/m

3 )C

once

ntra

tion

(µg/

m3 )

2

1.5

1

0.5

03/31 6/30 9/30

16

12

8

4

03/31 6/31 9/30

Observations CMAQ

(a)

(c)

Region II

Region IX

Fig. 2. Time series of the mean measured sulfate concentrations at I

estimates in four regions—biweekly moving averages.

4.2. Geographic and seasonal subdivisions

A single RMSE value can conceivably character-ize the whole simulation. However, since we wish todetermine when and where the quality of CMAQand REMSAD estimates significantly differ fromeach other, we avoided this fierce averaging andorganized simulation results into ten geographicalareas and 12 seasons (months) prior to RMSEcalculations. Two RMSEs, characterizing each ofthe models, respectively, were calculated for eachcombination of region and month by incorporatingall pairs of relevant observations and model values.The every 3-day sampling schedule at IMPROVEsites led to the following sample sizes for the monthsfrom January through December 2001: 11, 9, 11, 10,10, 10, 10, 11, 10, 10, 10, 9. Similarly, CASTNetmonthly sample sizes (based on weekly samplingdurations) were: 4, 4, 5, 4, 4, 4, 5, 5, 4, 4, 3. Onlysites with fewer than 15% missing values were used.

Ten geographic regions (Fig. 1) were identified inthe continental US on the basis of the modes ofvariation observed for aerosol sulfate (Irwin et al.,2004). These regions also reflect the broad natural,

10

7.5

5

2.5

03/31 6/30 9/30

6

4.5

3

1.5

03/31 6/30 9/30

estimates REMSAD estimates

(b)

(d)

Region VII

Region X

MPROVE sites and the corresponding CMAQ and REMSAD

Page 6: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESS

CMAQ more accurate than REMSAD

REMSAD more accurate than C MAQ

Apr il May

June July

August September

Fig. 3. Dentification of the most accurate model for sulfate

simulation during the 6 months of high sulfate concentrations—

IMPROVE network.

E. Gego et al. / Atmospheric Environment 40 (2006) 4920–4934 4925

topographic and climatic features encountered. Thearea extending from Texas north to the Nebraska-South Dakota border is not represented in thisstudy because it contains only one monitoring site.Table 1 recapitulates the number of observationsites operated by CASTNet and IMPROVE in eachregion.

4.3. Model-to-model comparison

To compare CMAQ and REMSAD performancesand determine whether or not they significantlydiffer from each other, we calculated the differencesbetween matched pairs of RMSEs, respectively,characterizing CMAQ and REMSAD in eachregion/month, and evaluated the mean of thesedifferences. If CMAQ and REMSAD skills aresimilar, the paired RMSEs should be about equaland the mean of their differences should therefore beclose to 0. If the skill of one model considerablysurpasses the other, the mean of the paireddifferences will be significantly different from zero.

The statistical test chosen to perform thiscomparison is the Wilcoxon matched-pairs signedrank test (WMP), a non-parametric counterpart ofthe matched-pairs Student t-test. The non-para-metric option releases us from concerns about thenormality of the underlying population of RMSEdifferences with only limited loss of power. As in apaired Student’s t test, the null hypothesis of theWMP test states that the mean of the pairedRMSEs differences is zero; i.e., the mean of RMSEscharacterizing CMAQ equals that of RMSEscharacterizing REMSAD. Unlike the Student t-test,WMP test is not performed on the RMSE valuesbut on their ranks.

Practically speaking, applying the WMP test firstrequires calculations of the differences betweenmatching RMSE values. The absolute values ofthese differences are then ranked from least togreatest, and the ranks are assigned the sign of thecorresponding RMSE difference. Finally, the sumsof the positive (T+) and the negative ranks (T�) arecalculated. If the performances of CMAQ andREMSAD are similar, the positive and negativerank-sums should be approximately equal(T+ffiT�). A large gap between the positive andnegative rank-sums indicates that one model per-forms consistently better than the other. The smallerof the two T values is selected as the test statistic(T 0) and the probability of observing a T 0 valueequal or smaller than that observed if the null

hypothesis is true (often simply referred to as theprobability level (p)) is identified. The null hypoth-esis is rejected (meaning that the qualities of the twomodels’ estimates are significantly different) if p

exceeds a threshold value, usually set at 5%.When assessing within-month performance con-

trasts, we use the means of the ten paired regionalRMSEs for a given month, calculate their differencesand submit these differences to the WMP test. Whenassessing performance contrasts within each region,we compare the 12 paired monthly RMSEs char-acterizing a given region. In addition, we compare thepaired RMSEs encountered during the 6 months withhighest concentrations, hence contrasting models skillat reproducing high concentration seasons.

5. Results

5.1. Aerosol sulfate

5.1.1. Comparison with improve observations

The RMSE values characterizing CMAQ andREMSAD ability to reproduce sulfate observations

Page 7: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–49344926

for each month and region are shown in Table 2.The calculated T 0 values in correspondence toinformation provided in Table 2 and the respectiveprobability levels are summarized in Table 3. Tofacilitate inspection of the results, tests that confirmmodel performances to be significantly different(po5%) are underlined and the model whosequality was found better (CMAQ or REMSAD) isidentified. Note that the probability levels shown inTable 3 are associated with each individual test;caution is advised if considering several teststogether.

Considering the 12 monthly RMSEs within eachregion, model performance could be differentiatedin regions II, IV, VIII, and IX but not elsewhere. Ofthese regions, CMAQ had a lower RMSE in all butregion II. When focusing only on the six months ofhigh concentrations (from April–September), sig-nificant differences between the two models arefound in regions II, III, IV, VII, and IX. Onceagain, CMAQ had a lower RMSE in all theseregions but region II.

Fig. 2 displays the time series of averagemeasurements and corresponding model estimates

Time (day)

Time (day)

Observations CMAQ

Con

cent

ratio

n (µ

g/m

3 )C

once

ntra

tion

(µg/

m3 )

2.5

2.0

1.5

1.0

0.5

03/31 6/30 9/30

3/31 6/30 9/30

12

9

6

3

0

(a)

(c)

Region II

Region VI

Fig. 4. Time series of the mean measured sulfate concentrations at

estimates in four regions—biweekly moving averages.

in four of the ten regions identified in the contiguousUS: region II (panel a); region VII (panel b), regionIX (panel c) and region X (panel d). Region II waschosen as representative of the western US, whilethe three eastern regions have the highest annualmean sulfate concentrations. IMPROVE data (24-hconcentrations) and corresponding model estimatesshow high short-term variability, a condition thatprevents their clear display. We therefore chose todisplay the temporal average of 5 successivesampling days, i.e. a quasi bi-weekly signal sincesampling events are separated by 3 days in theIMPROVE protocol, rather than individual obser-vations. Temporal averaging attenuates the ex-tremes, therefore, improving visualization. Eachline in Fig. 2 corresponds to the bi-weekly signalof the mean observations and model estimates at allsites in the region displayed.

Although the largest in the western US, sulfateconcentrations in region II (panel a) are modest incomparison to those observed in the eastern USInterestingly, CMAQ and REMSAD signals arevery close but neither resembles the patternobserved there. Yet, REMSAD daily estimates

estimates REMSAD estimates

3/31 6/30 9/30

12

9

6

3

0

3/31 6/30 9/30

12

9

6

3

0

(b)

(d)

Region VI

Region IX

CASTNet sites and the corresponding CMAQ and REMSAD

Page 8: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–4934 4927

(not displayed but substituted for by the bi-weeklysignals) were often slightly closer to reality thanCMAQ’s, justifying why the WMP test showedREMSAD to be superior. In region X, both modelsfollow the same pattern, but the amplitudes ofCMAQ fluctuations more closely resemble those ofthe observations. Results are similar in region VII(panel b), where CMAQ predictions seem moreaccurate than REMSAD’s during the high concen-trations season but less accurate during the lowconcentration seasons. That explains why the WMPtest pertaining to the entire year did not showCMAQ to be better while the test performed on thehigh concentration months only (6 monthly RMSEvalues) did.

In inspecting model performance by month, itappears that April, July and August yielded con-trasted differences, each time in favor of CMAQ.Complementing this comparison, Fig. 3 indicatesthe model (CMAQ or REMSAD) that led to themost accurate results (smallest RMSE value) for

Table 4

Comparison of CMAQ and REMSAD estimates of sulfate concentrati

Month Region

I II III IV V

RMSE (mg m�3) characterizing CMAQ estimates, by month and region

Jan. 0.37 0.17 0.24 0.76 0.46Feb. 0.35 0.40 0.32 1.00 1.54Mar. 0.37 0.59 0.41 0.70 0.82Apr. 0.34 0.59 0.31 1.36 0.74May 0.39 0.95 0.43 0.76 0.60Jun. 0.56 0.94 0.41 0.99 0.53Jul. 1.21 1.26 0.39 0.89 0.30Aug. 0.79 1.12 0.52 0.89 0.76Sep. 0.53 0.90 0.37 0.65 0.45Oct. 0.26 0.83 0.25 0.53 0.50Nov. 0.29 0.25 0.21 0.56 0.33Dec. 0.23 0.13 0.25 0.43 0.26

RMSE (mg m�3) characterizing REMSAD estimates, by month and reg

Jan. 0.36 0.17 0.23 0.76 0.69Feb. 0.21 0.28 0.22 0.99 1.55Mar. 0.36 0.61 0.37 0.75 0.88Apr. 0.23 0.63 0.44 1.45 0.79May 0.49 1.02 0.53 0.83 0.60Jun. 0.57 1.01 0.44 1.08 0.56Jul. 1.33 1.30 0.52 1.09 0.26Aug. 0.90 1.17 0.61 1.00 0.81Sep. 0.60 0.97 0.53 0.93 0.58Oct. 0.24 0.85 0.36 0.73 0.54Nov. 0.26 0.33 0.33 0.95 0.49Dec. 0.24 0.12 0.08 0.46 0.33

each region duting the 6 months of the highconcentrations season.

5.1.2. Comparison with CASTNet observations

Fig. 4 displays the temporal evolution of sulfatein regions II, VI, VII and IX; the three latteridentified as the most polluted areas in theCASTNet network. Because the weekly averageconcentrations measured by CASTNet do not showthe short-term variability present in IMPROVEdata, each graph displays raw information and not amoving average of several sampling events. Asobserved with IMPROVE data, both models con-stantly underestimate sulfate concentrations inregion II. Elsewhere, CMAQ seems better thanREMSAD at simulating high concentrations peri-ods.

Table 4 summarizes the RMSEs calculated foreach month and region while the T 0 values andcorresponding probability levels are indicated inTable 3. With CASTNet data being the basis for

ons to observations at CASTNet sites

VI VII VIII IX X

1.41 1.21 1.04 1.55 1.51

1.68 1.36 0.98 1.71 1.44

1.23 1.10 0.40 0.86 1.28

1.05 1.60 0.38 0.85 1.06

1.60 2.05 1.36 1.55 1.33

1.29 0.91 0.85 1.37 1.19

1.88 2.01 0.77 1.72 2.14

1.41 2.49 1.22 2.40 1.96

0.74 1.16 0.84 1.61 1.33

0.68 1.20 0.66 1.35 1.11

0.58 0.83 0.32 0.56 0.82

0.81 0.73 0.51 0.77 0.61

ion

1.49 1.25 0.70 1.60 1.44

1.15 0.94 0.33 0.74 0.99

0.95 0.78 0.42 0.63 1.21

1.30 2.10 0.64 1.10 1.43

1.66 2.88 1.42 1.82 1.81

1.55 2.51 0.99 2.54 1.87

2.58 4.55 1.25 3.33 2.70

2.52 3.96 1.62 4.96 2.78

1.20 2.24 1.11 1.81 0.99

0.37 0.45 0.75 0.62 0.66

0.69 1.03 0.47 0.47 0.76

0.71 0.57 0.63 0.86 0.57

Page 9: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–49344928

comparison rather that IMPROVE’s, CMAQ esti-mates proved better in regions II, IV, VII and VIII.Of these four regions, only region VII experienceshigh sulfate concentrations. In examining themonths from April–September, CMAQ estimatesseem more realistic than REMSADs everywhere butin regions I and V where the quality of the twomodels could not be differentiated. Note thatCASTNet maintains only three and one site,respectively, in regions V and I.

The WMP test applied to sulfate estimates sortedby month showed that CMAQ was a better modelduring the 6 months of high concentrations, i.e.,from April to September, corroborating the regio-nal contrasts during this period, and again duringNovember. CMAQ estimates were more accuratethan REMSADs in all ten regions for the months ofApril, May and August, as proven by the null valuesof T 0.

5.1.3. Differences between the evaluation results

obtained with IMPROVE and CASTNet.

As stated in Section 2, the frequency and durationof a sampling event are quite different in theIMPROVE and CASTNet protocols. While IM-

Observations CMAQ

Con

cent

ratio

n (µ

g/m

3 )C

once

ntra

tion

(µg/

m3 )

2

1.5

1

0.5

03/31 6/30 9/30

Time (day)

3/31 6/30 9/30

4

3

2

1

0

Time (day)

(a)

(c)

Region II

Region VII

Fig. 5. Time series of the mean measured nitrate concentrations at I

estimates in four regions.

PROVE data describe 24-h average concentrationsmeasured every 3 days, CASTNet data are 7-dayintegrated samples. Because of these differences,IMPROVE and CASTNet data allow assessment ofdifferent model skills: the ability of a model toreproduce day-to-day variations or longer-term(weekly) variations. In the case of sulfate, CMAQskills show better than REMSADs when assessed bycomparison with CASTNet data but not so notablywith IMRPOVE data. This finding tends to showthat CMAQ edge over REMSAD resides in itsability to better reproduce changes in weeklyaverage but not the day-to-day fluctuations. Thepreceding comments need to be considered cau-tiously, since differences in the location and thenumber of sites per region for each network mayalso explain these results.

5.2. Aerosol nitrate

5.2.1. Comparison with improve observations

Fig. 5 presents the temporal evolution of aerosolnitrate in regions II, VI, VII and IX. Generallyspeaking, air quality in the western US is better thanin the East. However, in terms of nitrate pollution,

estimates REMSAD estimates

3/31 6/30 9/30

5.0

3.75

2.5

1.25

0

3/31 6/30 9/30

5.0

3.75

2.5

1.25

0

(b)

(d)

Region VI

Region IX

MPROVE sites and the corresponding CMAQ and REMSAD

Page 10: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–4934 4929

region II ranks as the third worst region, just afterregions VII and IX. As seen on panel a, measuredconcentrations in region II do not diminish assharply during late spring and summer as in theother three regions. CMAQ and REMSAD over-predict this concentration decrease and thereforeunderestimate the extent of nitrate pollution inregion II during summertime. Both models simulateregion VI (panel b) reasonably well, althoughREMSAD largely exaggerates late fall concentra-tions. In region VII (panel c), both models over-predict high concentrations but are faithful toobservations during the low concentration season.Finally, CMAQ and REMSAD overestimate con-centrations in region IX during most of the year,especially during the high concentrations period(cold season).

Table 5 summarizes the RMSEs that characterizethe goodness of CMAQ and REMSAD nitrateestimates for each month and region. Table 6 showsthe corresponding T 0 and probability levels. Inanalyzing their year-long performance, it appears

Table 5

Comparison of CMAQ and REMSAD estimates of nitrate concentrati

Month Region

I II III IV V

RMSE (mg m�3) characterizing CMAQ estimates, by month and region

Jan. 1.70 2.40 1.16 0.70 1.05Feb. 0.99 1.92 0.86 0.34 0.97Mar. 0.31 1.50 0.41 0.38 0.67Apr. 0.70 1.81 0.39 0.41 0.98May 0.41 1.10 0.19 0.20 0.22Jun. 0.31 0.76 0.11 0.24 0.29Jul. 0.22 1.01 0.08 0.18 0.20Aug. 0.47 0.67 0.11 0.16 0.15Sep. 0.34 0.94 0.12 0.16 0.24Oct. 0.22 1.70 0.39 0.33 0.41Nov. 0.92 3.05 0.30 0.46 0.75Dec. 0.79 0.63 0.51 0.59 1.49

RMSE (mg m�3) characterizing REMSAD estimates, by month and reg

Jan. 1.72 2.32 1.03 0.65 0.91Feb. 1.02 2.05 0.73 0.33 0.78Mar. 0.32 1.49 0.44 0.40 0.62Apr. 0.61 1.13 0.27 0.31 0.86May 0.29 1.03 0.18 0.20 0.27Jun. 0.13 0.69 0.11 0.23 0.23Jul 0.23 0.87 0.09 0.18 0.18Aug. 0.21 0.69 0.12 0.17 0.15Sep. 0.35 0.80 0.12 0.14 0.28Oct. 0.24 1.73 0.16 0.14 0.33Nov. 1.10 3.08 0.36 0.43 0.98Dec. 0.68 0.65 0.38 0.49 1.27

that CMAQ and REMSAD did not simulateregions IV, VII nor X equally well. More specifi-cally, REMSAD was significantly better at simulat-ing region IV, an area with extremely lowconcentrations, while CMAQ failed at reproducingthe general pattern observed. On the other hand,CMAQ simulated regions VII and X more faith-fully. Fig. 6 and panel c of Fig. 5 details observedand simulated temporal evolution of nitrate in theareas with contrasted model performances. Carry-ing out the WMP test on results describing themonths of high concentration only (from Novemberto April) led to similar results, although the super-iority of CMAQ in region VII could no longer beproven.

As illustrated in Fig. 7, which indicates the mostaccurate model for each region during the 6 monthsof high-nitrate concentrations, the performances ofREMSAD and CMAQ are close. The WMP testapplied to results sorted by month (Table 6)showed that only April and November led tosignificantly different model performance, with

ons to observations at IMPROVE sites

VI VII VIII IX X

2.55 1.30 0.81 1.55 0.71

0.47 1.37 0.94 2.02 0.85

0.91 1.75 0.85 2.28 1.13

0.96 1.86 1.40 2.68 0.73

0.22 0.57 0.34 1.01 0.40

0.10 0.41 0.12 0.51 0.39

0.22 0.31 0.13 0.67 0.32

0.11 0.26 0.17 0.39 0.35

0.13 0.78 0.15 0.84 0.23

0.73 1.33 0.85 1.67 0.37

1.73 2.51 1.07 2.70 0.60

1.87 1.28 0.30 1.10 0.46

ion

1.77 1.58 0.88 1.58 1.62

0.45 1.70 0.43 1.74 1.11

1.18 1.90 0.94 2.23 2.03

0.40 1.55 1.04 1.90 0.68

0.26 0.47 0.25 1.18 0.43

0.40 0.68 0.26 1.85 0.45

0.16 0.34 0.13 1.41 0.46

0.10 0.67 0.19 1.82 0.40

0.14 0.98 0.28 1.55 0.44

0.96 1.43 0.95 1.44 0.43

2.75 2.60 1.57 2.54 1.02

1.13 2.39 1.47 2.21 0.97

Page 11: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESS

Table 6

Differentiation of the performances of CMAQ and REMSAD to simulate aerosol nitrate with Wilcoxon matched pair test (statistic T 0—

probability levels p less than 5% are underlined)

Season Network Region

I II III IV V VI VII VIII IX X

Within each region

All year IMPROVE T 0 31 23 24 11 21 38 13 23 25 4

p (%) 28.47 11.67 13.31 1:34R

8.81 48.49 2:12C

11.67 15.06 0:17C

CASTNet T 0 15 25 13 32 20 27 30 18 23 34

p (%) 3:20R

15.06 2:12C

31.10 7.57 19.02 25.93 5.49 11.67 36.67

High conc. months IMPROVE T 0 9 10 3 2 6 8 4 7 7 1

p (%) 50.0 450. 7.81 4:69R

21.88 34.38 10.94 28.13 28.13 3:13C

CASTNet T 0 6 10 5 7 2 6 6 7 9 6

p (%) 21.88 50.00 15.63 28.13 4:69C

21.88 21.88 28.13 42.19 21.88

Network Month

Jan. Feb. Mar. Apr. May. Jun. Jul. Aug. Sep. Oct. Nov. Dec.

Within each month

MPROVE T 0 25 24 13 0 23 17 22 12 12 26 7 21

p (%) 42.29 38.48 8.01 0:10R

34.77 16.11 31.25 6.54 6.54 46.09 1:86C

27.83

CASTNet T 0 27 7 16 11 19 20 18 9 3 18 27 11

p (%) 50.00 1:86R

13.77 5.27 21.58 24.61 18.75 3:22C

0:49C

18.75 50.00 5.27

C: indicates that CMAQ is significantly better than REMSAD; R: indicates that REMSAD is significantly better than CMAQ.

Con

cent

ratio

n (µ

g/m

3 )

0.8

0.6

0.4

0.2

03/31 6/30 9/30

Time (day)

Observations CMAQ estimates REMSAD estimates

3

2.25

1.5

0.75

03/31 6/30 9/30

(a) (b)

Region IV Region X

Fig. 6. Time series of the mean measured nitrate concentrations at IMPROVE sites and the corresponding CMAQ and REMSAD

estimates in the Southwest and the South-Atlantic regions—bi-weekly moving averages.

E. Gego et al. / Atmospheric Environment 40 (2006) 4920–49344930

REMSAD estimates more accurate in April andCMAQ’s more appropriate in November.

5.2.2. Comparison with CASTNet observations

Table 7 shows the RMSE characterizing nitrateestimates by region and month, CASTNet observa-tions being used as the basis for comparison.

Analyzing all monthly results with the WMP test(Table 6) showed that models performance wasdistinguishable only in regions I and III, regionswith relatively low nitrate levels, with REMSADmore accurate in region I and CMAQ moreaccurate in region III. Inspection of results pertain-ing to the 6 months of high concentrations indicated

Page 12: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESS

January February

March April

November December

CMAQ more accurate than REMSAD

REMSAD more accurate than C MAQ

Fig. 7. Identification of the most accurate model for nitrate

simulation during the 6 months of high observed nitrate

concentrations—IMPROVE network.

E. Gego et al. / Atmospheric Environment 40 (2006) 4920–4934 4931

that region V, sampled by a single CASTNetmonitor, is the only region where model perfor-mance was different, with REMSAD better thanCMAQ. Also illustrating this absence of contrast,Fig. 8 presents the evolution of nitrate concentra-tions in the four regions previously depicted: regionsII, VI, VII and IX. As already observed withIMPROVE data, CMAQ and REMSAD predic-tions are too low in region II during the summer-time (panel a). Both models reproduce regions VIand VII quite well with fairly the same degree ofinaccuracy, justifying why WMP test did notdistinguish their respective skills. Finally, predic-tions in region IX (panel d) are overestimated byboth models, especially during the months of highconcentrations.

Examination of model performance for eachmonth with the WMP test (Table 6) shows thatREMSAD was better than CMAQ for simulation ofFebruary (month with high concentrations), whileCMAQ was more accurate for August and Septem-ber, both months with low concentrations. Model

performances could not be differentiated for the 9other months.

Generally speaking, whereas CMAQ was oftenshown superior for simulation of aerosol sulfate, itsskill at reproducing nitrate seems to be comparableto that of REMSAD.

6. Summary

Ironically, while the modeling community devotesa great amount of attention and energy to rigorousquantification of the relevant chemical and physicalprocesses, the results of a model evaluation areoften communicated in qualitative terms. State-ments such as ‘the model is doing fairly well’ or ‘themodel has been greatly improved’ are often madewithout quantitative supporting evidence. Subjec-tivity can even be more treacherous when judgingthe relative performance of two models. When canone conclude that a model is better than another?Does a simple visual examination of model outputssuffice? Must one model prove superior for all timesand points in the model domain?

The US EPA has endeavored to facilitate themodel-to-model comparison of CMAQ and RE-MSAD models by performing an annual simulationof air quality over the contiguous US using bothmodels driven by identical inputs (meteorology,emissions, etc.). Here we attempted to make bothqualitative and quantitative assessments of therespective skills of CMAQ and REMSAD tosimulate aerosol nitrate and sulfate. Graphs ofobserved and modeled time series obtained withCMAQ and REMSAD were compared. In rareoccasions, a visual examination of these graphs wassufficient to decide upon the best model, such aswhen one of them provided estimates systematicallycloser to observations than the other one. In mostcases, the prevalence of one model over the otherappeared weak, transient and/or local. In theseinstances, to remove any subjectivity from ourinterpretation, we calculated a standard evaluationmetric (the RMSE) to characterize the goodness ofeach model and submitted matched pairs of thisevaluation metric to a statistical test of comparisonof means (Wilcoxon Matched-Pairs signed ranktest). In an effort to unveil the areas and timeperiods where the quality of CMAQ and REMSADestimates significantly differ from each other,simulation results were organized into ten geogra-phical areas and monthly periods for calculation ofthe RMSEs. The WMP test was used to determine

Page 13: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESS

Table

7

ComparisonofCMAQ

andREMSAD

estimatesofnitrate

concentrationsto

observationsatCASTNet

sites

Month

Region

III

III

IVV

VI

VII

VIII

IXX

RM

SE

(mg

m�

3)

chara

cter

izin

gC

MA

Qes

tim

ate

s,by

month

and

regio

n

Jan.

0.36

1.15

0.22

0.36

0.68

2.86

1.08

0.70

1.41

1.43

Feb.

0.37

0.53

0.42

0.35

1.28

1.46

0.81

1.31

1.48

1.18

Mar.

0.41

1.40

0.12

0.38

0.70

1.69

1.42

0.91

1.93

1.42

Apr.

0.80

0.73

0.33

0.45

0.51

1.20

0.83

1.23

2.57

1.45

May

0.59

0.84

0.17

0.41

0.33

0.87

0.45

0.40

1.14

1.03

Jun.

0.27

0.90

0.19

0.43

0.29

0.57

0.26

0.12

0.51

0.78

Jul.

0.47

1.05

0.17

0.44

0.20

0.49

0.14

0.16

0.48

0.64

Aug.

0.50

0.95

0.22

0.37

0.40

0.38

0.20

0.19

0.31

0.76

Sep.

0.46

1.10

0.14

0.33

0.36

0.54

0.36

0.20

0.54

0.68

Oct.

0.29

1.51

0.19

0.36

0.36

0.67

1.20

1.23

1.45

1.24

Nov.

0.27

1.01

0.16

0.45

0.67

2.06

1.69

1.31

2.27

1.94

Dec.

0.21

0.31

0.19

0.26

0.84

2.21

0.92

0.46

1.05

1.01

Jan.

0.53

0.94

0.23

0.27

0.41

2.44

1.33

1.19

1.37

1.66

RM

SE

(mg

m�

3)

chara

cter

izin

gR

EM

SA

Des

tim

ate

s,by

month

and

regio

n

Feb.

0.30

0.54

0.39

0.34

1.14

1.54

0.63

0.99

1.24

1.06

Mar.

0.31

1.44

0.18

0.34

0.59

1.71

1.69

1.11

1.93

1.61

Apr.

0.64

0.58

0.37

0.63

0.63

0.91

0.76

0.93

1.87

1.32

May

0.52

0.92

0.29

0.47

0.39

1.04

0.37

0.36

1.04

1.11

Jun.

0.31

0.92

0.21

0.45

0.26

1.16

0.42

0.39

1.25

0.74

Jul.

0.32

1.05

0.17

0.44

0.12

0.75

0.25

0.36

0.89

0.62

Aug.

0.22

0.97

0.23

0.37

0.40

0.69

0.21

0.62

1.10

0.78

Sep.

0.41

1.10

0.19

0.35

0.43

0.90

0.44

0.48

1.23

0.79

Oct.

0.27

1.55

0.20

0.39

0.45

1.31

0.93

1.39

1.54

1.12

Nov.

0.32

1.04

0.21

0.46

0.32

2.08

1.57

1.37

2.28

1.84

Dec.

0.15

0.33

0.17

0.22

0.55

1.30

1.43

1.46

2.05

1.37

E. Gego et al. / Atmospheric Environment 40 (2006) 4920–49344932

Page 14: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESS

Con

cent

ratio

n (µ

g/m

3 )C

once

ntra

tion

(µg/

m3 )

2

1.5

1

0.5

03/31 6/30 9/30

Time (day)

3/31 6/30 9/30Time (day)

5

4

3

2

1

03/31 6/30 9/30

5

4

3

2

1

0

3/31 6/30 9/30

8

6

4

2

0

Observations CMAQ estimates REMSAD estimates

(a) (b)

(c) (d)

Region II Region VI

Region VII Region IX

Fig. 8. Time series of the mean measured nitrate concentrations at CASTNet sites and the corresponding CMAQ and REMSAD

estimates in four regions.

E. Gego et al. / Atmospheric Environment 40 (2006) 4920–4934 4933

the significance of the differences between CMAQand REMSAD performances during each monthsimulated and within each region.

The results of this analysis can be summarized asfollows:

In the case of sulfate, significant differences in thequality of CMAQ and REMSAD estimates werefound for about half the tests performed (for 3–7months out of the 12 months of simulation,depending on the observation network consid-ered, and in three to seven of the ten regionsindividualized, depending on the observationnetwork and the length of the simulated period(all year vs. 6 months of high concentrations).When differences between CMAQ and RE-MSAD proved significant, it was almost exclu-sively in favor of CMAQ. The exception to thatstatement is region II (California) where RE-MSAD estimates match IMPROVE data moreclosely. CMAQ was shown significantly better inthree to four regions, whether assessed withIMPROVE or CASTNet data and better at

reproducing all six months of high concentra-tions when compared with CASTNet data.CMAQ superiority was not as prevalent,although existent, if assessed with IMPROVEobservations, leading us to speculate that thestrength of the CMAQ model does not reside inits ability to simulate day-to-day variations butthe longer-term (weekly) fluctuations.

� In the case of nitrate, significant differences in the

quality of CMAQ and REMSAD estimates werefound in less than 20% of the tests performedwith no model performing consistently better.For instance, if using the IMPROVE data as thebasis for comparison, REMSAD was found abetter model for simulating region IV butCMAQ simulation of region X was morefaithful. Similarly, while February was bettersimulated by REMSAD, CMAQ proved betterfor reproducing the CASTNet data of Augustand September. As a result, the only statementwe consider fair concerning simulation of nitrateis that both models seem to perform reasonablyand equally well.

Page 15: An objective comparison of CMAQ and REMSAD performances

ARTICLE IN PRESSE. Gego et al. / Atmospheric Environment 40 (2006) 4920–49344934

Acknowledgements

This research was partially funded by the USDepartment of Commerce through contracts withDr. E. Gego (EA133R-03-SE-0710), with the Uni-versity of Idaho to Dr. P. S. Porter (EA133R-03-SE-0372), and with the State University of New York toDr. C. Hogrefe (EA133R-03-SE-0650). The viewspresented are those of the authors and do not reflectthe views or policies of the US Department ofCommerce.

References

Byun, D.W., Ching, J.K.S. (Eds.), 1999. Science algorithms of the

EPA Models-3 Community Multiscale Air Quality Model

(CMAQ) modeling system. EPA/600/R-99/030, US Environ-

mental Protection Agency, Office of Research and Develop-

ment, Washington, DC, 20460.

Carolina Environmental Programs, 2003. Sparse Matrix Opera-

tor Kernel Emission (SMOKE) Modeling System, University

of Carolina, Carolina Environmental Programs, Research

Triangle Park, NC.

Gego, E., Porter, P.S., Irwin, J.S., Hogrefe, C., Rao, S.T., 2005.

Assessing the comparability of ammonium, nitrate and sulfate

concentrations measured by three air quality monitoring

networks. Pure and Applied Geophysics 162, 1919–1939.

Grell, G.A., Dudhia, J., Stauffer, D., 1994. A description of the

fifth-generation Penn State/NCAR Mesoscale Model (MM5).

NCAR Technical Note, NCAR/TN-398+STR.

Hanna, S.R., 1994. Mesoscale meteorological model evaluation

techniques with emphasis on needs of air quality models. In:

Pielke, R.A., Pearce, R.P. (Eds.), Mesoscale Modeling of

theAtmosphere. Meteorological Monographs, vol. 25. Amer-

ican Meteorological Society, Boston, MA, pp. 47–58.

ICF Consulting, 2002. User’s Guide to the Regional Modeling

System for Aerosols and Deposition (REMSAD) Version 7.

Systems Applications International/ICF Consulting, San

Rafael, CA 94903, 153pp.

Irwin J.S., Gego E., Hogrefe C., Jones J. M., Rao S.T., 2004.

Comparison of sulfate concentrations simulated by two

regional-scale models with measurements from the IM-

PROVE network, Ninth Conference on Harmonization

within Atmospheric Dispersion Modeling for Regulatory

Purposes, Garmish-Partenkirchen, Germany.

McNally, D., 2003. Annual application of MM5 for calendar

year 2001, prepared for US EPA by D. McNally, Alpine

Geophysics, Arvada, CO, 2003, 179pp.

US Environmental Protection Agency, 2003. User’s guide to

MOBILE6.1 and MOBILE6.2; EPA Office of Air and

Radiation, EPA420-R-03-010, Assessment and Standards

Division, Office of Transportation and Air Quality, US

Environmental Protection Agency, 262pp.

Further reading

Eder, B., Yu, S., 2004. A performance evaluation of the 2004

release of MODELS-3 CMAQ. Preprints of the 27th NATO/

CCMS International Technical Meeting on Air Pollution

Modeling and Its Applications, Banff, Canada, pp. 166–173.