Quality Control - General• Quality control in data assimilation is used to ensure that the analysis
is not degraded by the inclusion of certain observations• Good quality control also often requires knowledge of the details the
instrument design and processing system• Quality control can be applied by removing observations from
analysis or by adjusting observation errors for observations– An infinite observation error in the same as removing an observation– Monitoring is enhanced if observation is used in analysis, but weight is zero.
• Quality control requires monitoring of data and has significant impacts on the design of the database.
• Quality control is extremely important, but not recognized.
Observation errors
• For random observation errors should be Gaussian.
• Reasons for non-random errors– Representativeness error– Inability of forward model to properly model observation– Instrument Difficulties
Representativeness error• Models do not resolve all scales of the atmosphere. • Observations often are point measurements• When structures are observed which cannot be
resolved – representativeness error.• Observation is correct – but does not represent the
structures in the analysis• Example – Radiosonde launched in local
thunderstorm used for large scale analysis
Inability of forward model to properly model observation
• If forward model transforming analysis variables into observation variables cannot properly simulate observation
• Example – cloudy radiances from satellite when only simulating clear radiances
• Example 2 – not including CO2 in retrieval process – errors created at scales of CO2 variation
• Observations correct, but errors introduced which can be aliased into analysis
Satellite data• Most simulation problems with satellite data come
from 5 sources:– Instrument problems.
• Inadequate characterization• Degrading instrument
– Inadequately measured/modelled components of forward model.
– Inadequately understood physics
Satellite radiances• IR cannot see through clouds.
– Cloud height can be difficult to determine – especially with mixed FOVs.
– Since measures deep layers, not many channels completely above clouds.
• Microwave impacted by clouds and precipitation but signal is smaller from thinner clouds and can be modeled.
• Surface emissivity and temperature characteristics not well known for land/snow/ice.– Also makes detection of clouds/precip. more difficult over these
surfaces.• Error distribution may be asymmetric due to clouds and
processing errors.
Instrument Difficulties
• If the instrument is fails – observations usually not good (by definition)
• Instrument design – ascending/descending aircraft– Non-Gaussian errors – Scatterometer ambiguity
• Insufficient sensitivity – GPS RO• Monitoring vital to detect problems • Surface data examples
Model BA exceeds limit for observation
Below, obs rejected by QC
Mismatch between top planetary boundarylayer
Instrument Difficulties
• If the instrument is fails – observations usually not good (by definition)
• Instrument design – ascending/descending aircraft– Non-Gaussian errors – Scatterometer ambiguity
• Insufficient sensitivity – GPS RO• Monitoring vital to detect problems • Surface data examples
Quality control usually has multiple steps
• Choice of observations used in analysis• Physically realistic?
– Pressure > 0, Moisture > 0, Surface observation < 500mb, etc.
• Complex quality control – attempting to save observations• Station monitoring – Blacklist
– Must have capability to add/remove stations from list
• Manual QC – Human intervention– Greatly complicates operational system
• Instrument specific checks based on instrument design• Variational QC (replaces buddy check)
Variational quality control• Modifies weight given observation based on deviation from
guess/analysis• Assumes that observation error is not Gaussian.• Examples – Assume error PDF is Gaussian + constant • Examples – Assume error PDF is sum of Gaussians• Will make minimization non-linear – will generally slow
convergence and may create multiple minima.• Generally makes weight a function of how close the data is
to current solution – solution should be close to final solution
Correlated errors• If observational errors are correlated – all parts of
quality control must be rethought• Nearby obs may have same signal because of real
signal or correlated errors.– Can be extremely difficult to distinguish correlated error
• Correlated errors occur because of– Instrument/forward model biases– Retrievals from multiple observations spreads errors
across all components of retrieval
Data Monitoring• It is essential to have good data monitoring. • Usually the NWP centres see problems with
instruments prior to notification by provider (Met Office especially).
• The data monitoring can also show problems with assimilation systems.
• Needs to be ongoing/real time.• https://groups.ssec.wisc.edu/groups/itwg/nwp/mon
itoring
Summary• Quality control is a very important (but under-
recognized) component of data assimilation• Often performed in multiple steps – platform dependent
and cross-platform components• Inclusion of quality control within analysis can be done
(variational QC) and is being done at operational centres.• However, variational QC significantly complicates
minimization• Much QC work not documented in literature
ReferencesAndersson, E., and H. Jarvinen, 1999: Variational quality control Quart. J. Reoy. Meteor. Soc., 125, 697-722 – See also ECMWF tech memo #250.
Baker, Nancy L., 1994: Quality control of meteorological observations at Fleet Numerical Meteorology and Oceanography Center, U.S. Navy, Naval Research Laboratory,23p.
Collins, W. G., 2001: The operational complex quality control of radiosonde heights and temperatures at the National Centers for Environmental Prediction. Part I: Description of the Method, Jour. of App. Meteor.,40.2, 137-151.
European Centre for Medium Range Weather Forecasts, WMO/ECMWF Workshop on Data Quality Control Procedures, Reading, England, March 6-10, 1989, Papers., Reading, England, European Centre for Medium-Range Weather Forecasts (ECMWF), 1989
Gandin, Lev S., 1988: Complex quality control of meteorological observations, Mon.Wea.Rev.116. 1137-1156.
Ingleby, Nbruce, Lorenc, Andrew C., 1993, Bayesian quality control using multivariate normal distributions, Quart. J. Roy. Met. Soc, 119, 1195-1225.
Kelly, G; Andersson, E; Hollingsworth, A; Loennberg, P; Pailleux, J; et al.; 1991: Quality control of operational physical retrievals of satellite sounding data, Mon. Wea. Rev.,1866-1880.
Lorenc, A.C., and Hammon, O., 1988: Objective quality control of observations using Bayesian methods – Theory, and a practical implementation Quart. J. Roy. Met. Soc. 114, 515-543.
Purser, R.J., 2011: Mathematical principles of the construction and characterization of a parameterized family of Gaussian mixture distributions suitable to serve as models for the probability distributions of measurement errors in nonlinear quality control, NCEP Office Note #468.
Various ECMWF lectures and training courses.