Resources on forecastevaluation/verification

Barbara BrownJoint Numerical Testbed

NCAR, Boulder, CO

July 2011

WMO Verification web pageDeveloped andmaintained byworking group onforecast verificationresearch

Includes FAQs,references and links totutorial presentations

http://www.cawcr.gov.au/projects/verification/

Verification discussiongroup:vx-discuss@rap.ucar.edu

EUMETCAL Online Learning Module

http://www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm

Free on-linetutorial withexercises

Includes modulesonCategoricalContinuousProbabilistic

Verificationmethods

Warner (2010):Numerical

BooksJolliffe and

Stephenson:Forecast

Verification - APractitioner’s

Guide inAtmospheric

SciencesPublished by Wiley

(new version in2011)

Weather andClimate

PredictionPublished by

CambridgeUniverstity PressIncludes chapter

Wilks (2011): StatisticalMethods in the AtmosphericSciencesPublished by ElsevierIncludes extensive chapter onverification methods

Forecast Evaluation Tools

R Statistics Package:Includes many statistical tools,including the R VerificationLibrary

MET is freely available andsupported to the community

Main focus: ModelverificationIncludes tools for point andgridded data, ensembleforecasts, spatial methodshttp://www.dtcenter.org/met/users/

Introduction to R

Tara JensenNCAR/RAL/JNT

What is R?

• A statistical programming language• In part, developed from the S ProgrammingLanguage from Bell Labs (John Chambers)

• Created to allow rapid development ofmethods for use in different types of data.

• Create new graphics. Many defaultparameters are chosen, but users retaincomplete control.

Why R?

• R has become the dominant language in the statisticalresearch community.

• R is Open Source and free.• Runs on all operatingsystems• Nearly 2,400 packages contributed.• Packagesand applicationsin nearly every field ofscience, business and economics.

• See R Notes, R Journal and Journal of StatisticalSoftware.www.jstatsoft.org

• More than 100 books with accompanying code• Very large, active user base.

Why not R?

• NCL, IDL, Matlab, SAS, … are all viablealternatives to R. If you are a part of an activecommunity of researchers using anotherlanguage, do likewise.

• If we were biostatisticians we would be usingSAS. Book Title: “Analyzing Receiver Operating Characteristic Curves with SAS”

• Consider building verification functions andutilities as part of code development .Verification need not be an external process toforecasting.

The R Community• Developers

– R Core Group (17 members), only 2 have left since1997

– Major update in April/October (freeze dates, betaversions, bug tracking, ...)

• Mailing lists– Help list ~ 150 messages/day, archived,searchable.

• 5 International Conferences, 2 US, 1 China

• Source code• Binary compilations (Windows, Mac OS, Linux• Documentation ( Main documents, plus numerouscontributed. Some in foreign languages.)

• Newsletter (replaced by R Journal.)• Mailing list (Several search engines)• Packages on every topic imaginable• Wiki with examples• Reference list of books using R. ( more than 100)• Task Manager

Everything about R is at www.r-project.org

Use R with scripts• In Linux - Emacs Speaks Statistics

– Providessyntax-based– Object name completion– Key strokeshort cuts– Commandhistory– Alt-x R to invoke R with Xemacs.

• In Windows, use editor– Added GUI features– <control>R sends a line or highlighted section into R.– Install package with GUIs– Save graphics by point and click.

• Mac OS– Similarto Windows with advantagesof system calls.

Coding principles

• Make verification code transparent and easyto read

• Comment and document liberally• Archive your code• Share your code• Label and save your data• Share your data

Packages in R

• Contributed by people world wide.• Allow scientists or statisticians to push theirideas.

• Apply and extend R capabilities to meet theneeds of specific communities.

• Accompany many statistical textbooks• Accompany applied articles (Adrian Raftery,Doug Nychka, Tilman Gneiting, Barbara Casati,Matt Briggs)

A sample of useful packages

•••••

verificationfields (spatial stats)radiosondesextRemesBMA(BayesianModelAveraging)

• BMAensemble• circular• Rsqlite

• Rgis, spatstat (GIS)• ncdf ( support fornetcdf files )

• Rcolorbrewer• randomForests

Packages

• Packages must be installedto call.• Packages must be called to use.• Base packages are installed by default.

10 most useful function in R

• aggregate - applies a function to groups ofdata subset by categories.

• apply - incredibly efficient in avoiding loops.Applies functions across dimensions of arrays.

• layout - creatively divide a print region.• xyplot (in the lattice package) slightly advancegraphic techniques

• %in% returns logical showing which elementsin A are in B. (e.g A%in%B)

More top 10

•

•

•

•

•

table – create contingency tabel counts.boot – apply bootstrap function correctlyread.fwf – read fixed width format datapar – control everything in a graphsystem( ) – allows you to call systemcommand from R

• pairs – the most under utilized plot – plots amatrix of 4 columns in a 4x4 plot layout

R Exercises

•

•

•

•

•

Choose groups of 3-4 – find a computerLog onto machinesBring up at least 2 xterms>cp /wrfhelp/MET/R-packages/example/* .>vi intro2R.2011comet.R

or>kwrite intro2R.2011comet.R &