Click here to load reader

Dieter Mayer, Reinhold Steinacker, Andrea Steiner University of Vienna, Department of Meteorology and Geophysics, Vienna, Austria Presentation at the 10th

  • View
    216

  • Download
    0

Embed Size (px)

Text of Dieter Mayer, Reinhold Steinacker, Andrea Steiner University of Vienna, Department of Meteorology...

Folie 1

VERA-QC, a new Data Quality Control based on Self-ConsistencyDieter Mayer, Reinhold Steinacker, Andrea SteinerUniversity of Vienna, Department of Meteorology and Geophysics, Vienna, Austria

Presentation at the 10th European Conference on Applications of Meteorology (ECAM)Berlin, 14 September 2011

1Outline

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Motivation for VERA-QCApplicability and basis of VERA-QCMathematical background of VERA-QCDeviations and error detectionHandling special station alignmentsConclusion and availability of VERA-QC

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Motivation for VERA-QC

High quality data is needed as input for VERA

What is VERA?Analysing observations to grid points (complex topography)Combining interpolation (TPS) & downscaling (Fingerprints)Features of VERAModel independentNo need for first guess fieldsWorks on real time & operational basisApplications of VERA & VERA-QCReal time model verificationBasis for nowcastingEvaluation of case & field studiesComputation of analysis ensembles

High quality data is needed as input for VERA

What is VERA?Analysing observations to grid points (complex topography)Combining interpolation (TPS) & downscaling (Fingerprints)Features of VERAModel independentNo need for first guess fieldsWorks on real time & operational basisApplications of VERA & VERA-QCReal time model verificationBasis for nowcastingEvaluation of case & field studiesComputation of analysis ensembles

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Selecting or designing a QC?Properties of VERA & its applicationsExisting QC-methods

Requirements toselect / design QC

Bayesian QCVariational QCQC using OIQC using IDQC using SRLimit checksInternal consistency checksmodel independentno back-ground fieldsmodel verificationreal timefast (not iterative)field studiesno statistical informationcomplex topographyhandle inhomogeneous station distributionanalysis ensemblespropose deviationsAnswer: there is a need for a new QC-method

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Applicability of VERA-QCBasis: spatio and / or temporal consistency of dataRequirement: High degree of redundancy in observations

Example:

VERA-Analysis for precipitation (green) & MSL-pressure (black)

Dots and stars:Observations for precip. & pressure

Depending on station density & scale of phenomenonExpressed as station distance and decorrelation length QC applicable if / >> 1 (GTS: pMSL,Q,Qe)

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Basis of VERA-QCError affected observations (rough) observation field YoCorrected observations (smoother) analysis field Ya = Yo + DYMain task is to receive deviations DY

Example: South-West to North East pressure-gradient with some artificial errors:

Note: DY is not a simple difference between observation and interpolation

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Mathematical Core of VERA-QCGoal: receive deviations to obtain smooth analysis field.

d1,d2, D: dimensionsn, N: grid points

P: prim. neighborsm,M: main stationss,S: second. neighbors

Defining cost function J as squared curvature of analysis field:

Curvature of analysis field Cya is not known Taylor series expansion:- Building global cost function: (taking into account all stations and grid points)- Solving optimization problem for deviations :

7

Questions regarding the cost function:Q1: Where should the cost function be evaluated? A1: Regular grid is too expensive, take station pointsQ2: What are main stations, primary and secondary neighbors? A2: m: Main station: one station after another s: (secondary) neighbors of m p: (primary) direct neighbors of m Q3: ? Which stations contribute to the Taylor series expansion? A3: A certain station and its natural neighbors. More than one station is allowed to be erroneous!

Concept of natural neighbors

Method connecting stations: Delaunay Triangulation

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Triangulation / Computing curvatures Typical example for realistic station distribution and Delaunay Triangulation

Defining local grids around stations Interpolate station values YS to grid points n:Computing curvatures

(Inverse distance interpolation)

Simplest example: 1D, 1 spikeOutlier corrected partially, butcounter swinging at neighbors Solution: correcting erroneous observation should reduce cost function. Compute weighted deviations:

with

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Weighting Deviations

Three possibilities to handle an observation

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Deviations and Gross ErrorsNo gross error Obs. correctedGross error Obs. rejectedNo gross error Obs. acceptedyesnoyes

noyesno

a, b and c: parameter dependent, user defined thresholdsVERA-QC is repeated without rejected observations

Error propagation possible at close by stationsExample: circles with stations, cluster in centerBoth stations obtain significant deviations Combine both stations to one fictive cluster stationCompute deviation for cluster station Add deviation to both stationsRepeat VERA-QC for modified observations

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Cluster Treatment

Properties of VERA-QC:Applicable to 1, 2, 3 and 4 dimensional problemsHigh efficiency in detecting errors compared to other QC methodsNo simple averaging algorithm Can handle very inhomogeneous station distributionsModel independent, fast, no iterations necessary Deviations can be stored to compute biasImplemented as Matlab stand alone application, runs on Server & PC

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.ConclusionsFurther Informations:Publication: Steinacker, R., D. Mayer, and A. Steiner 2011, Data Quality Control Based on Self Consistensy. Accepted in Monthly Weather Review.Poster Presentation: A. Steiner, Operational Application of VERA-QC, Challenges and how to cope with them. Poster Hall, Thursday 16-17:00.

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.Availability of VERA-QC Homepage: http://www.univie.ac.at/amk/veraflex/test/intern/

VERA-QC is freely available for non-commercial use

The EndAcknowledgments: Austrian Science Fund (FWF), support under grant number P19658Contact: [email protected]

http://www.univie.ac.at/amk/veraflex/test/intern/Thank you for your attention

Is VERA-QC an averaging technique?Considering a signal at only 3 stations (unlikely to be a gross error)Unweighted deviations smooth signalWeighted deviations only soften contrast

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.

VERA-QC in higher dimensions

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.

Interpolate irregularly distributed station values to regular grid (Thin plate spline)Downscaling with the help of idealized physically motivated patternsVERA in a nut shell

10th European Conference on Applications of Meteorology (ECAM)Berlin, 12-16 September 2011IMG ViennaMayer et.al.

Solution

Unexplained field

Explained fieldWeight

Fingerprint