A STATISTICAL VALIDATION OF THE SNOWPACK MODEL IN A MONTANA CLIMATE

Christopher C. Lundy·, Robert L. Brown and Edward E. AdamsDepartment of Civil Engineering, Montana State University-Bozeman

Karl W. BirkelandUSDA Forest Service National Avalanche Center

Michael LehningSwiss Federal Institute for Snow and Avalanche Research

ABSTRACT: Recently, a computer model has been developed by the Swiss Federal Institute for Snowand Avalanche Research that- simulates the evolution of a natural snow cover. Using commonmeteorological parameters as input, SNOWPACK predicts characteristics such as snowpack temperatureand density, in addition to snow microstructure and layering. An investigation was conducted to evaluatethe effectiveness of SNOWPACK in a Montana climate. A weather station was constructed in the BridgerMountains near Bozeman, Montana, to provide the meteorological parameters necessary to runSNOWPACK. Throughout the 1999-2000 winter, weekly snow profiles were performed in undisturbedsnow to provide a benchmark for the model output. Density, grain size. and crystallography wererecorded on 10 cm intervals over the full snow depth, and the temperature profile was monitored with athermocouple array. Finally, the meteorological parameters were input into SNOWPACK, and a statisticalcomparison was performed comparing the predicted snowpack to the observational data. Snowpacktemperatures are predicted reasonably accurately by SNOWPACK. The modeled and observed densitiescorrelated well, but the model typically underestimates snowpack settlement. Comparison of grain sizeand shape was problematic due to different definitions utilized by the model and observer, but stilldemonstrated some agreement

KEYWORDS: snowpack modeling, snow metamorphism, model validation, snowpack evolution

1. INTRODUCTION

Recent advances in snow research, as well as theincreasing availability of powerful computersystems, have led to the development of computermodels that are becoming better at predicting theevolution of a mountain snowpack. The moreadvanced models have already been usedoperationally and provide avalanche hazardforecasters and other mountain safety experts withyet another tool for evaluating the alpinesnowpack.

The focus of this paper is SNOWPACK. anu~erical snow cover model developed by theSWISS Federal Institute for Snow and AvalancheResearch. SNOWPACK is a predictive finiteelement model that computes the heat and masstransfer, stresses, and strains within the snowpack

-*CorreSp<?~ding author address: Christopher C.bU~dY, CIVIl Engineering Dept., Montana State4~~~rsity, .Bozeman, MT 59717; tel: (406) 763-

, em~ll: lundy@mcn.net

259

(Lehning et a/., 1998). SNOWPACK uses commonmeteorological parameters as inputs and provideas output predicted snowpack temperature.density, grain and bond size, and coordinationnumber. The model also characterizes theexpected crystal type based on the grain. shapeparameters sphericity and dendricity (Figure 1).Currently, wet snow metamorphism and surfacehoar prediction are still under development, andthe models for dry snow metamorphism are beingimproved (Lehning et a/., 1998).

One problem associated with efforts to model theevolution of a mountain snowpack is the lack of ameaningful comparison of the model predictions toan actual snowpack. Plots of simulated andmeasured snowpack parameters are abundant inthe literature, but simply comparing these graphsvisually does not provide a meaningful orconsistent evaluation of the similarity betweenobserved and predicted values. Introducingstatistical methods using well-establishedmeasures provides a means of quantifying theaccuracy of the models.

The field research site is located approximately!one kilometer north of the Bridger Bowl Skinear Bozeman, Montana, in an area knownWolverine Basin. This region falls withinintermountain or transition climate regimedefined by McClung and Schaerer (1993). ftfraverage annual snowfall of approximately 6.5 m .measured at the adjacent ski area. The site islarge, open, relatively flat meadow situated at anelevation of 2240 m (Figure 2).

A weather station was constructed at the site toobtain the necessary input data for theSNOWPACK model. From 17 November 1999t06April 2000, the following were collected on 30minute intervals:

To complete an objective validationSNOWPACK, a weather station was consto provide the meteorological paranecessary to run the model. During the 1999-2winter, regular snow profiles were conductedprovide a benchmark for the model outputmeteorological data was then inputSNOWPACK, and a "predicted" snowpackcomputed. Finally, the use of statistical methallow a thorough and objective comparison ofmodel output to field observations.

3. METHODS

3.1 Description of Field Site

dd=O, sp=l

Sphericity varies from 0 to 1..

dd=O, sp=O

Figure 1. Relation of the model parameterssphericity and dendricity to the ISCI symbols(Colbect et al., 1990).

dd=l, sp=.5oB.....

To date, validation of the snowpack simulationmodels has not been adequately addressed usingan objective and numerical approach. This willcontinue to limit their improvement andacceptance, since few users are willing to expendthe resources necessary to operationalize asnowpack model without first having access to anextensive evaluation of the program.

2. PURPOSE

Figure 2. Aerial photo showing location of Wolverine Basin weather station.

260

3.2 Collection of Snow Profile Data

Additionally, a temperature-measurement arraywas utilized to obtain a real-time temperatureprofile within the snowpack. It was constructedtram a 3 m PVC tube fitted with thermocouples on5 cm intervals, standing vertically with the bottomthermocouple at ground level. The PVC tube isfilled with foam so that the entire unit has a lowthermal conductivity.

••••••

Air temperatureSnow surface temperatureRelative humidityWind speedReflected shortwave radiationTotal snow depth.

Before any comparison of the numeric data can beundertaken, predicted model data must becalculated at the same depths within thesnowpack as the snowpit observations (i.e. on 10cm intervals). To accomplish this task, a techniquewas devised by (Lehning et al., this volume) toobtain model results at desired locations. First, todevelop a basis for comparing the predicted andobserved snowpack heights, even if the totaldepths differ, a normalization. of depth isperformed. Next, linear interpolation is appliedbetween normalized heights to obtain modelresults at depths that correspond with theobserved measurements. Since grain type. is notmeasured on a continuous numeric scaleinterpolation is not possible and the only feasibl~technique is to use the grain type value thatoccurs at the height closest to the observedlocation.

On a weekly basis, snowpits were excavated inundisturbed, fenced-off snow near the instrumenttower. Density was measured with a triangulardensity box of known volume, and weighed on aportable digital scale. Grain diameter and the ISCIgrain classification (Colbeck et al., 1990) weredetermined with a 20x Pentax hand lens.Observations were recorded every 10 cm throughthe full depth of the snowpack.

3.3 Comparison of the Predicted Snowpack tothe Observed Snowpack

While SNOWPACK includes a graphical userinterface that presents a visual description of thesnowpa~k predicted by the model, a simple visualcomp~nson of the model results to the snowpitdata IS not adequate to objectively evaluate themodel. A better method is to employ familiarstatistical measures to evaluate the level ofagreement between the modeled and observedparameters:

3.4 Statistical Measures of Model Performance

A detailed summary of statistical descriptors thatevaluate a. model's ability to match anobservational dataset is provided in Imam et a/.(1999). These goodness-of-fit indicators fallloosely into one of two categories: residual-basedand. ~tatistical association-based. By employingstatistical measures from both categories, a morecomplete description of the model's performanceis obtained.

In the residual-based category, the mean bias (B)and the root mean square error (RMSE) arecommonly used for model verification (Sorooshianet al., 1983; Imam et a/., 1999). The mean biasindicates the direction of the expected model errorand is a useful measure of a model's tendencytowards overestimation or underestimation. Incontrast, the RMSE estimates the expectedmagnitude of error associated with a model'sprediction.

The model itself was configured to output thesn~w profile data for each day at 1100 hours,Which .corresponded with the time at which the:OWPlts were typically performed in the field. It is

IS p~Qfile data that is compared to the weeklysnowplt observations.

••••

TemperatureDensityGrain sizeGrain type.

Among indicators of statistical associationPearson's correlation coefficient, r, is perhaps th~most common. The correlation coefficient has anupper bound of 1, indicating perfect positive linearcorrelation, and a lower bound of -1corresponding to negative linear correlation:Although Pearson's r is familiar among manyscientists and often used for validation purposes,other researchers have provided argumentsagainst its use for model verification (Imam et al.,1999; Imam, 1994; Fox, 1981; Willmott, 1981).Another measure of association is the Nash­Suttcliffe coefficient of efficiency, E, which is upper

261

Table 1. Statistical measures comparing the predicted and observed snowpack parameters.

262

4. RESULTS AND DISCUSSION

-d0.9S0.760.38

E0.770.300.05

r0.90

0.850.30

0

-0-2~

e.a -4 0l!GlQ,

E -6GlI- 0 00"C -8Ql

0c=Gl

! -100

Figure 3. Observedtemperature.

-12 -JL-,....-----,----,--,----.-----,

-12 -10 -8 -6 -4 -2 0Modeled Temperature ,e)

3It is apparent from the RMSE of 69.15 kg/mSNOWPACK has some diffiCUlty predicting 5snow cover density. A mean bias B of -48.1kg/m3 confirms that the majority of the error:

4.2 Density

reasonably well. The RMSE is only 0.97 °C,the mean bias B shows very little tendency toover or underestimation. Since the temperatumeasurements are accurate to within ± 0.5 °C,RMSE demonstrates that the model does ajob of predicting temperature.

The measures of association rand dare neaone and the coefficient of efficiency E is fairly highsuggesting a high degree of correlation betweeltthe observed snowpack temperatures and thoSepredicted by the model. Referring to Figure 3.observed-predicted data pairs lie very close to1:1 line which represents perfect agreement Itshould also be noted from Figure 3 that there~several instances where SNOWPACK predictsisothermal temperatures, but colder temperatureswere measured. The accuracy of the model alsOtends to diminish as the temperatures decrease.

RMSE

0.97°C69.15 kg/rn3

0.42mm

B0.10°C

-48.15 kg/rn3

-0.08 rnrn

n196

177179

TemperatureDensityGrain Size

By all statistical measures, the SNOWPACKmodel predicts the snow cover temperatures

Table 1 summarizes the results of the descriptivestatistical analysis for the numerical parameterstemperature, density, and grain size. While thereare no hard-and-fast rules for interpreting thesemeasures, the utilization of several differentstatistical descriptors allows meaningfulconclusions to be drawn regarding theperformance of the SNOWPACK model.Additionally, the analysis prOVides a consistentframework for comparing different versions of themodel, or evaluation of the model using datasetsfrom different seasons or geographical locations.

The preceding discussion serves to emphasizethat there is no single statistical descriptor that willeffectively assess a model's ability to predictobserved data. Only by combining the merits ofseveral different measures can a complete modelevaluation be obtained.

4.1 Temperature

bounded by 1 and yields higher magnitudes· withincreasing model accuracy. The coefficient ofefficiency can also assume negative values, whichhave a less intuitive interpretation. To addressthis, Willmott and Wicks (1980) proposed an indexof agreement, d, which is bounded by [0 1] so thatit does not produce negative values.

The preceding statistical indicators are applicableto numeric parameters such as temperature,density, and grain size, but not to grain type whichis measured on a categorical scale. Instead,Cramer's Phi (V) and Sakoda's adjustedcontingency coefficient (C*) (Agresti, 1996) areused which quantify the degree of associationbetween two categorical variables. These Chi-Square (X2 )-based coefficients are convenientsince they range from 0 to 1 with higher valuesindicating better association. Both V and C* arecomputed twice; once for the majority grain typeF1, and again for the minority classification F2.

1:1500

- E"'s 400 .5.a ell

N~ en~300 c:" 0 ";c: ...II C)

~ 200 'CellII~~ CDII)II

,Q! 10000

00 100 200 300 400 500

Modeled Density (kg/m3)

1:1

3 0

2 <Ill

1

o

O..JL---.----,-----r-o 1 2 3

Modeled Grain Size (mm)

Figure 4. Observed versus modeled density.

resent in the model's prediction is due to a~nsistent underestimation <:>f the measuredsnowpack density. Figure 4.lllustrates that theobserved-predicted data pairs are somewh.atcentered on the 1:1 line until the observed densItyapproaches 250 kg/m3

. For densities greater than250 kg/m3

• the data diverges markedly from ~he1:1 line as the model increasingly underpredlctsthe measured density.

Despite the discrepancy in the magnitudes of ~he

predicted and observed density, the correlationmeasures yielded more positive results. Thecoefficient of efficiency E is fairly low at 0.30. but rand dare fairly high at 0.85 and 0.76. respectively.The high r shows that there is a tendency toward alinear relationship; however, a look at Figure 4reveals that the linear relationship does not followthe 1:1 line. This means that the general theory ofthe model may be sound. but that an adjustmentof parameters may be necessary to improve theaccuracy of density prediction.

4.3 Grain Size

The comparison of predicted and observed grainsize was difficult and does not supply conclusiveinformation about the accuracy of the model. All ofthe statistical measures for grain size given inTable 1 give low values. The RMSE is large at0.42 mm. but shows little or no bias with Bequaling -0.08 mm. The statistical associationSCOres r, E, and d are well below acceptablevalues.

Figure 5. Observed versus modeled grainsize.

The problem in this comparison lies partly in thediffering definitions of grain size used by the modeland the field observer. The model chooses 0.6mm as the size of new snow particles, and allowsonly the growth of these grains. As a result, it isevident from Figure 5 that SNOWPACK neverpredicts a grain diameter less than 0.6 mm,. but onseveral occasions. grain sizes less than thiS werereported in the field observations. Furthermore.grain sizes greater than 1 mm are sel.dompredicted by the model, but were routlne.lyobserved in the field. SNOWPACK uses a graindiameter that is independent of thecrystallography; essentially all grains are treatedas spheres. This is not true of the field .observer.whose estimation of grain size is often tied to theshape of the snow grain. especially for facet~d

crystals. The estimation of grain size in the field. ISunavoidably a subjective measurement. Despitethese contrasting definitions of grain size an.d .thepossibility of observer error. the low statisticalmeasures suggest the need for model refinementin this area.

4.4 Grain Tvpe

Cramer's V for both F1 and F2 are both low andindicate only a weak correlation between thepredicted and observed values (Table 2). Theadjusted contingency coefficients C* of 0.71 and0.66 suggest a somewhat large~ a~ou~t <:>fassociation is present, but the relationship IS stilinot very strong.

263

264

5. CONCLUSION

Table 2. Statistical associationmeasures for the majority andminority grain types.

Since snow crystallography is not measuredcontinuous or numeric scale, the use of astatistical measures is required. The resultsstatistical association measures demoweak correlation between the modeledobserved grain types. Like grain size.shape is a SUbjective observation;developing a classification scheme thatboth utilized by the model and ameasured in the field would be advantageous

This analysis does not evaluate theprediction of surface hoar since we inactivroutine during model execution. This portionmodel is still under development (Lehning1998) and reportedly overpredicts surfaceoccurrence (Pielmeier et al., 2000). Anotherof the model needing improvement is wetmetamorphism. Currently, SNOWPACK haSrudimentary provisions for simulating wetmetamorphism. Since this routine influgrain size, crystal shape, and density, itSare included in the analysis. Future workwet snow capabilities of SNOWPACKimportant for ablation prediction, hydpurposes, and for applying the model tomaritime climates.

temperatures, and additional diffiCUltymodeled temperatures approach 0 °C. Theof the model to effectively simUlatetemperature is crucial since mostprocesses are strongly temperature depe

Though snowpack density is predictedsuccessfully than temperature, two of themeasures of statistical correlation givevalues. The data also demonstrates that thesignificantly underpredicts the actualdensity when the densities exceeds akg/m3

. Still, our results provide infomight be useful for improving density pred"future versions of the model.

A meaningful comparison of predictedobserved grain size is difficult due to d"definitions of grain size and the sub' "human measurement. The statisticalgave generally poor results and indicatecorrelation between the simulated andvalues. Therefore, the results of theindicate not only the need to improve thebut also the utility of a more staobservation technique using a definition ofsize similar to that employed by SNOWPAI

206 0.41 0.71206 0.34 0.66

n V C·Majority (F1 )Minority (F2)

The statistical measures utilized in the analysisindicate that the SNOWPACK model predicts thetemperature profile within the snow cover fairlyaccurately. However, inspection of the plot ofpredicted and measured temperature reveals adiminishing model accuracy with colder

Similar to grain size, SNOWPACK uses analternate grain classification scheme, namelydendricity and sphericity. Since there is little basisfor estimating these parameters during fieldobservation, the model must choose a standardISCI grain shape according to variouscombinations of dendricity and sphericity.Obviously, this is not an ideal or particularlyaccurate procedure. A more desirable techniquewould be to develop a common classificationsystem employed by both the SNOWPACK modeland the observer. Another alternative is to focusless on the crystallography, and more on themicrostructural parameters that actually define thephysical properties of the snow. Crystal type isobserved in the field primarily as an indicator ofthe degree of bonding and strength possessed bythe snow. For instance, when rounded grains areencountered, a high degree of bonding andstrength is assumed; the opposite is true whenfaceted crystals are observed. If a reliable fieldmethod was developed for measuring bond size,and perhaps bond density, the classification ofgrain shape may be less important.

From 17 November 1999 to 6 April 2000,meteorological data were collected from amountain weather station adjacent to Bridger BowlSki Area near Bozeman, MT. During the sameperiod, full snowpack profiles were performed on aweekly basis within a short distance from theweather station. By running the SNOWPACKmodel using the collected weather data and.comparing the output to the snow profiles, athorough evaluation of the predictive capabilities ofthe model was possible. Statistical tests wereutilized to make the comparison objective andconsistent.

· esent form, the SNOWPACK model canIn itS pr a useful tool for avalanche forecasters~:er practitioners who need to know theand roes and structure of the snow cover, but do~tways have ~e ability .to conduct frequentnot rofiles in a given location. Of course, userssttf1II ~odel must understand the limitations of theof the t version of SNOWPACK. Future model~rre~ements and validation should increase theiIIIproll accuracy of SNOWPACK and itsavera t't'usefulness as a tool for snow prac I loners.

5. REFERENCES

Adams E. E. and Sato, A. 1993. Model foreff~tive thermal conductivity of a dry snowcover composed of uniform ice spheres. Ann.Glaciology 18, 300-304.

Agresti,A. 1996. Int~uction to Categorical DataAnalysis. John Wiley and Sons, New York.

Box, G. E. P., Hunter, W. G., and Hunter, J. S.1978. Statistics for Experimenters. John Wileyand Sons, New York.

Colbeck, S. C, Akitaya, E., Armstrong, R., Gubler,H., Lafeuille, K., Leid, D., McClung, C., andMorris, E. 1990. The internationalclassification for seasonal snow on theground. International Commission on Snowand Ice, 23 pp.

Fox, D. G. 1981. Judging air quality modelperformance: A summary of the AMSworkshop on dispersion model performance.BUll., Am. Met8Orol. Soc. 62, 599-609.

Imam, B., Sorooshian, S., Mayr, T., Schaap, M.,Wosten, H., and Scholes, B. 1999.Comparison of pedotransfer functions tocompute water holding capacity using the vanGenuchten model in inorganic soils. IGBP-D/SWorking Paper #22.

Imam, B: 1994. Non-linear uncertainty analysis formultiple criteria natural resource decisionsupport system. Ph.D. Dissertation, Universityof Arizona.

lehning, M., Bartelt, P., Brown, R. L., Russi, T.Stockli, U., and Zimmerli, M. 1998. A networkof automatic weather and snow stations andsupplementary model calculations providingsnowpack information for avalanche warning.

265

In Proc. 1998 International Snow Sci.Workshop, Sun River, Oregon, 225-233.

McClung, D. and Schaerer, P. 1993. TheAvalanche Handbook. The Mountaineers,Seattle, Washington. 271 pp.

Nash, J. E. and Suttcliffe, J. V. 1970. River flowforecasting through conceptual models, Part I- A discussion of principles. J. Hydrol. 10,282-290.

Pielmeier, C., Schneebeli, M., and Stucki, T. 2000.Snow texture: a comparison of empiricalversus simulated texture index for alpinesnow. Ann. G/aciol. 32, in press.

Sorooshian, S., Gupta, V. K., and Fulton, J. L.1983. Evaluation of maximum likelihoodparameter estimation techniques forconceptual rainfall runoff models: influence ofcalibration data variability and length on modelcredibility. Water Resources Research, 19(1),251-259.

Willmott, C. J. 1981. On the validation of models.Phys. Geography 2,184-194.

Willmott, C. J. and Wicks, D. E. 1980. An empiricalmethod for the spatial interpolation of monthlyprecipitation within California. 1, 59-73.