Developments at the US GDAC

ADMT #10 – Toulouse, France30 September - 2 October 2009

Mark Ignaszewski

FNMOC

ADMT-10 30 September – 2 October 2009

Actions Completed

• Automated Greylist submissions

• Implemented Technical File v2.3 format checking

• Implemented new “latest data” file handling– Real-time and delayed-mode profiles in separate files

– Files directly in “latest-data” (no year/month directories)

– Files being limited to roughly 100Mb

– Running in parallel with old method until GDACs are consistent and users are informed.

ADMT-10 30 September – 2 October 2009

Actions Pending

• Automate file removal – Dec ‘09

• MD5 signature – Nov ‘09

– Will start when new format checker goes “live”

• New Format Checking (separate agenda item)

ADMT-10 30 September – 2 October 2009

Enhanced Profile Checking

• Meta-data checks• Date Checks• QC code checks• <PARAM> variable checks• D-mode file checks

ADMT-10 30 September – 2 October 2009

Meta-data Checks

• PLATFORM_NUMBER:

– 5 or 7 numeric digits (second digit “9” for 7)

• DATA_STATE_INDICATOR:

– One of the recommended codes from reference table 6

• DIRECTION: ‘A’ or ‘D’

• DATA_CENTRE: Valid for the DAC

• DATA_MODE: ‘A’, ‘D’, or ‘R’• INSTRUMENT_REFERENCE: Set (Warning)

• POSITIONING_SYSTEM: Set (Warning)

• WMO_INST_TYPE: Valid (ref table 8) (Warning)

ADMT-10 30 September – 2 October 2009

Date Checks

• All dates are checked for validity and consistency

• DATE_UPDATE and DATE_CREATION must be set

• String date settings checked for validity

– DATE_UPDATE / DATE_CREATION / HISTORY_DATE / CALIBRATION_DATE

– 14 digit strings; valid (e.g., seconds must be 0 to 59)

• Consistency checks

ADMT-10 30 September – 2 October 2009

Consistency of Dates

ADMT-10 30 September – 2 October 2009

• All dates must be after Jan 1, 1997 and before the file time

• JULD and JULD_LOCATION within 1 day

• No specific ordering among HISTORY_ and CALIBRATION_DATE

• On file submittal by DAC: DATE_UPDATE within 2 days of time submitted. (WARNING)

• Needed to allow for time zone “fudge factor”

QC Code Checks

• JULD_QC and POSITION_QC: Valid values

ADMT-10 30 September – 2 October 2009

<PARAM> Checks

• STATION_PARAMETER:– Only valid parameter names– No “blank” entries (WARNING)– No duplicate entries (WARNING)– PRES, TEMP, PSAL are required

• Check that <PARAM> variables exist for every STATION_PARAMETER

• Check that no other <PARAM> variables with data exist in the file.

• If mode = ‘A’ or ‘D’: Check that all <PARAM>_ADJUSTED have data– Subject to the rules in the QC manual (i.e. QC = ‘4’)

ADMT-10 30 September – 2 October 2009

<PARAM> Checks (continued)

• <PARAM>_QC and _ADJUSTED_QC– Only valid QC codes - No “fill values”

– Missing data flagged with 0, 4, 9

– Real-time profiles: Only codes 0 through 4

– Required parameters (PRES, TEMP, PSAL):• Cannot be code 0 if data is not missing

• PROFILE_<PARAM>_QC: Check value

• Check that N_PARAM and N_LEVEL are not larger than necessary

ADMT-10 30 September – 2 October 2009

D-mode File Checks

• DATA_MODE = “D”

• DATA_STATE_INDICATOR = “2C” or “2C+”

• Same <PARAM> in PARAMETERS as in STATION_PARAMETERS

• If PRES_ADJUSTED_QC = 4, TEMP_ADJUSTED_QC and PSAL_ADJUSTED_QC = 4– *_ADJUSTED = missing

ADMT-10 30 September – 2 October 2009

D-mode File Checks

• SCIENTIFIC_CALIB_COMMENT and CALIBRATION_DATE set– for every <PARAM> and N_CALIB

• At least one HISTORY record

• HISTORY_INSTITUTION and _DATE set

• Plus, the <PARAM> and date checks previously discussed

ADMT-10 30 September – 2 October 2009

Results: Warnings

• 9 DACs have problems with NULLs in strings– KORDI seems to be OK

– Some only in few variables, some in many variables

• A couple “N_LEVELS too large”– KORDI sets the variable larger than necessary a lot

• 1 “INSTRUMENT_REFERENCE” not set (JMA)

ADMT-10 30 September – 2 October 2009

Results: Date checks

• JULD after DATE_CREATION– Coriolis – only a few files

– INCOIS – many files – large time differences

• HISTORY_DATE and/or CALIB_DATE after DATE_UPDATE– CSIO – Many files – Big time differences

• Invalid dates:– AOML, INCOIS: Bad values

– MEDS: Too short (missing seconds)

ADMT-10 30 September – 2 October 2009

Results: <PARAM> checks

• ‘A’ or ‘D’: *_ADJUSTED not set – Identified some missing *_ADJUSTED data

– Also found some issues with the tests

• PROFILE_<PARAM>_QC: Check values– AOML, Coriolis, CSIO, JMA, MEDS

• Missing variables– CSIRO, INCOIS

• <PARAM>_QC and *_ADJUSTED_QC– Numerous inconsistencies reported

– Some illegal values

ADMT-10 30 September – 2 October 2009

Results: D-file checks

• DATA_STATE_INDICATOR: Coriolis and MEDS: – Question about “2C+”

– A few MEDS files set to “2B”

ADMT-10 30 September – 2 October 2009

Results: D-file checks

PARAMETER and SCIENTIFIC_CALIBRATION_*• PARAMETER or CALIB_DATE not set for many files

– PRES, TEMP: AOML, CSIO, MEDS

– TEMP, CNDC: CSIRO

– PRES, TEMP, PSAL: JMA

• SCI_CALIB_COMMENT not set for many files– TEMP, CNDC: CSIRO

– PRES, TEMP: JMA

• N_CALIB larger than necessary in many files– BODC (many files), Coriolis (few files)

• No calibration information in some D-files: Coriolis

ADMT-10 30 September – 2 October 2009

Plan

• Implement new checking in “DAC test”: 15 Oct 2009

• Implement new checking in routine processing AS ADVISORY: end Oct 2009

• Transition to IFREMER: Oct 2009

• Implement as operational checker: Nov 2009

ADMT-10 30 September – 2 October 2009