Quality control

John DerberNCEP/EMC

Reference slide at end

Quality Control - General• Quality control in data assimilation is used to ensure that the analysis

is not degraded by the inclusion of certain observations• Good quality control also often requires knowledge of the details the

instrument design and processing system• Quality control can be applied by removing observations from

analysis or by adjusting observation errors for observations– An infinite observation error in the same as removing an observation– Monitoring is enhanced if observation is used in analysis, but weight is zero.

• Quality control requires monitoring of data and has significant impacts on the design of the database.

• Quality control is extremely important, but not recognized.

Observation errors

• For random observation errors should be Gaussian.

• Reasons for non-random errors– Representativeness error– Inability of forward model to properly model observation– Instrument Difficulties

Observation errors

• For random observation errors should be Gaussian.

Representativeness error• Models do not resolve all scales of the atmosphere. • Observations often are point measurements• When structures are observed which cannot be

resolved – representativeness error.• Observation is correct – but does not represent the

structures in the analysis• Example – Radiosonde launched in local

thunderstorm used for large scale analysis

Inability of forward model to properly model observation

• If forward model transforming analysis variables into observation variables cannot properly simulate observation

• Example – cloudy radiances from satellite when only simulating clear radiances

• Example 2 – not including CO2 in retrieval process – errors created at scales of CO2 variation

• Observations correct, but errors introduced which can be aliased into analysis

Satellite data• Most simulation problems with satellite data come

from 5 sources:– Instrument problems.

• Inadequate characterization• Degrading instrument

– Inadequately measured/modelled components of forward model.

– Inadequately understood physics

Satellite radiances• IR cannot see through clouds.

– Cloud height can be difficult to determine – especially with mixed FOVs.

– Since measures deep layers, not many channels completely above clouds.

• Microwave impacted by clouds and precipitation but signal is smaller from thinner clouds and can be modeled.

• Surface emissivity and temperature characteristics not well known for land/snow/ice.– Also makes detection of clouds/precip. more difficult over these

surfaces.• Error distribution may be asymmetric due to clouds and

processing errors.

Observation Weight after QC

Observation Weight after QC

Observation Weight after QC

Observation Weight after QC

Instrument Difficulties

• If the instrument is fails – observations usually not good (by definition)

• Instrument design – ascending/descending aircraft– Non-Gaussian errors – Scatterometer ambiguity

• Insufficient sensitivity – GPS RO• Monitoring vital to detect problems • Surface data examples

Model BA exceeds limit for observation

Below, obs rejected by QC

Mismatch between top planetary boundarylayer

Instrument Difficulties

• If the instrument is fails – observations usually not good (by definition)

• Instrument design – ascending/descending aircraft– Non-Gaussian errors – Scatterometer ambiguity

• Insufficient sensitivity – GPS RO• Monitoring vital to detect problems • Surface data examples

Surface pressure data examples

Quality control usually has multiple steps

• Choice of observations used in analysis• Physically realistic?

– Pressure > 0, Moisture > 0, Surface observation < 500mb, etc.

• Complex quality control – attempting to save observations• Station monitoring – Blacklist

– Must have capability to add/remove stations from list

• Manual QC – Human intervention– Greatly complicates operational system

• Instrument specific checks based on instrument design• Variational QC (replaces buddy check)

Variational quality control• Modifies weight given observation based on deviation from

guess/analysis• Assumes that observation error is not Gaussian.• Examples – Assume error PDF is Gaussian + constant • Examples – Assume error PDF is sum of Gaussians• Will make minimization non-linear – will generally slow

convergence and may create multiple minima.• Generally makes weight a function of how close the data is

to current solution – solution should be close to final solution

Correlated errors• If observational errors are correlated – all parts of

quality control must be rethought• Nearby obs may have same signal because of real

signal or correlated errors.– Can be extremely difficult to distinguish correlated error

• Correlated errors occur because of– Instrument/forward model biases– Retrievals from multiple observations spreads errors

across all components of retrieval

Data Monitoring• It is essential to have good data monitoring. • Usually the NWP centres see problems with

instruments prior to notification by provider (Met Office especially).

• The data monitoring can also show problems with assimilation systems.

• Needs to be ongoing/real time.• https://groups.ssec.wisc.edu/groups/itwg/nwp/mon

itoring

Quality Monitoring of Satellite DataAIRS Channel 453 26 March 2007

Increase in SDFits to Guess

Summary• Quality control is a very important (but under-

recognized) component of data assimilation• Often performed in multiple steps – platform dependent

and cross-platform components• Inclusion of quality control within analysis can be done

(variational QC) and is being done at operational centres.• However, variational QC significantly complicates

minimization• Much QC work not documented in literature

ReferencesAndersson, E., and H. Jarvinen, 1999: Variational quality control Quart. J. Reoy. Meteor. Soc., 125, 697-722 – See also ECMWF tech memo #250.

Baker, Nancy L., 1994: Quality control of meteorological observations at Fleet Numerical Meteorology and Oceanography Center, U.S. Navy, Naval Research Laboratory,23p.

Collins, W. G., 2001: The operational complex quality control of radiosonde heights and temperatures at the National Centers for Environmental Prediction. Part I: Description of the Method, Jour. of App. Meteor.,40.2, 137-151.

European Centre for Medium Range Weather Forecasts, WMO/ECMWF Workshop on Data Quality Control Procedures, Reading, England, March 6-10, 1989, Papers., Reading, England, European Centre for Medium-Range Weather Forecasts (ECMWF), 1989

Gandin, Lev S., 1988: Complex quality control of meteorological observations, Mon.Wea.Rev.116. 1137-1156.

Ingleby, Nbruce, Lorenc, Andrew C., 1993, Bayesian quality control using multivariate normal distributions, Quart. J. Roy. Met. Soc, 119, 1195-1225.

Kelly, G; Andersson, E; Hollingsworth, A; Loennberg, P; Pailleux, J; et al.; 1991: Quality control of operational physical retrievals of satellite sounding data, Mon. Wea. Rev.,1866-1880.

Lorenc, A.C., and Hammon, O., 1988: Objective quality control of observations using Bayesian methods – Theory, and a practical implementation Quart. J. Roy. Met. Soc. 114, 515-543.

Purser, R.J., 2011: Mathematical principles of the construction and characterization of a parameterized family of Gaussian mixture distributions suitable to serve as models for the probability distributions of measurement errors in nonlinear quality control, NCEP Office Note #468.

Various ECMWF lectures and training courses.