GODAE Quality Control Pilot Project

GODAE Quality Control Pilot ProjectGODAE Quality Control Pilot Project

Status Report August 2007

IGST Meeting

Department Fisheries Ocean

St. John’s, New Foundland, Canada

Outline

1. New Data Sources

2. Output File and Reference Data

3. Plans and Issues

4. Argo FSI Float QC Outcomes

New Data Sources

• UKMO– long list form of UKMO QC output now available in real-time– UKMO back filled long list data to 2003 – thanks to Matt Martin– long list provides additional QC outcomes and both in situ and

potential temperatures (short list had only potential temperatures)– convention is to use in situ temperature in QC databases

• in situ calculated from potential when salinity reported• not always the case in the UKMO short list files

• BMRC– pushing QC data from BoM system to GODAE server– currently have data for 2005 and 2006 (nothing from 2007 yet) – working on BMRC data decoder


New Output File Format and Reference Data

• Call sign database conversion to NetCDF format complete– based on Coriolis/Argo multiple profile NetCDF file structure – easily expanded with new QC variables or new QC data providers– sorted by observation time and data center for ease of use

• Navy GDEM climate and climate variability profiles added as reference profiles in call sign database– temperature and salinity climatology

– global, monthly, 0.25 degree resolution

– 78 vertical levels (47 in upper 1000 m of water column)

– climate mean and standard deviations interpolated to observed levels for month in which observation taken


New MEDS QC Variable in Call Sign Database • decoded MEDS hex bit codes on QC tests performed

Meds QC Tests (binary string bit position from the RIGHT): Bit # - Test 0 - Platform Identification 1 - Impossible Date/Time 2 - Impossible Location 3 - Position on Land 4 - Impossible Speed 5 - Impossible Sounding 6 - Global Impossible Parameter Values 7 - Regional Impossible Parameter Values 8 - Increasing Depth 9 - Profile Envelop 10 - Constant Profile 11 - Freezing Point 12 – Spike 13 - Top and Bottom Spike14 – Gradient 15 - Density Inversion 16 - Levitus Seasonal Clim17 - Emery and Dewar Clim 18 - Asheville Clim 19 - Levitus Monthly Clim 20 – Waterfall 21 - Cruise Track 22 – Profiles23 – Bottom 24 - Temperature Inversion

• experience indicates only two tests flagged as typically performed– Levitus seasonal climate check and temperature inversion


Plans • initialize call sign databases with 2003-2006 production run

– NetCDF call sign databases will be hosted on US GODAE server– target date for completion Oct 2007

Issues• US GODAE server hardware upgrade has never taken place

– IBM X-346 machines plus 30 TB storage purchased years ago by ONR– never installed due to security issues – completion date still unknown

• GODAE QC application cannot run on existing server hardware– cannot do real-time updates of call sign databases as planned– in process of purchasing additional disk space for personal Linux

workstation to support GODAE QC function– using my Linux workstation (at best) only monthly updates of NetCDF call

sign databases on GODAE server possible


FSI Argo Floats Pressure Offset Errors

• discovered in February, 2007 that Argo profiles from SOLO floats with FSI CTD (Argo Program WHOI) have incorrect pressures – profile data may be offset upward by one or more pressure levels,

resulting in a significant cold bias for the data– a data denial list was posted on the Argo web site (287 WMO IDs)– data no longer distributed on GTS

• Argo science team asked if bad FSI float data had been accepted into operational assimilation systems

• FSI float QC outcomes from the NRL/FNMOC real-time QC system were evaluated from 2004 through 2006– instrumentation error checks– climate background field check– cross validation (buddy) check


float track

error probability of entire profile

(reject if p > 0.95)

GDEM climate anomaly

cross validation anomaly

error probability of profile level

(reject if p > 0.95)

NRL FSI Float QC Outcomes Display

Summary NRL/FNMOC QC of FSI Argo Float Pressure Errors

• rejected 11% of the float temperature profiles

• rejected 5% of the float temperature levels

• 185 float profiles rejected due to inadequate vertical sampling as a result of the vertical level rejections (created gaps in profiles)

Positive feedback from Howard Freeland on call sign databases

• From: Freeland, Howard [mailto:[email protected]] Sent: Wednesday, August 01, 2007 2:58 PMTo: Cummings, James [[email protected]]Subject: Rejected floats

I think the call sign files are exactly what I'd like to look at. I've been doing some simple investigations about data being rejected by less sophisticated users than GODAE people and have some interesting results...


END

dac source of data (FNMOC, NAVO, MEDS, Coriolis, UKMO, BMRC)

dtg observation time

dtg_qc quality observation time

lat, lon observation position

pos_qc quality observation position

n_lvl number levels

lvl observation depth levels

sal observed salinity

sal_fcst salinity forecast

s_fcst_err salinity forecast error

sal_clm salinity climatology

s_clm_err salinity climate variability

sal_qc salinity profile quality

Call Sign Database Structure Call Sign Database Structure

sal_qc_lvl salinity level quality

sal_flg salinity QC test failures

tmp observed temperature

tmp_fcst temperature forecast

t_fcst_err temp forecast error

tmp_clm temperature climate

t_clm_err temp climate variability

tmp_qc temperature profile quality

tmp_qc_lvl temperature level quality

tmp_flg temperature QC test failures

sign observation call sign

upd_dtg profile update time at center

qc_conflict flag indicating QC outcomes differ among data centers

NRL MEDS CoriolisUKMO

GODAE QC Application Program Interface (API) read different QC data formats, check for valid call sign, match

profiles (lat, lon, time, call sign), populate database structure with observation, QC codes, QC flags, and supplemental information, sort by observation time, update

NetCDF call sign database

1900305 Argo Float

A8ES9 Ship XBT

52007 TAO Mooring

Call Sign Databases

BMRC

Documents

GODAE Quality Control Pilot Project