Upload
cleary
View
57
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Datzilla Usage NCDC Perspective. Karsten Shein NOAA National Climatic Data Center [email protected]. NWS Sub-regional data stewardship workshop – Fargo, ND 12-13 May 2007. What is Datzilla?. - PowerPoint PPT Presentation
Citation preview
http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Datzilla UsageDatzilla UsageNCDC PerspectiveNCDC Perspective
Karsten SheinKarsten Shein
NOAA National Climatic Data [email protected]
NWS Sub-regional data stewardship workshop – Fargo, ND 12-13 May 2007
22 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
What is Datzilla?What is Datzilla?
Datzilla is a Web-based tool to report and track errors in NOAA-held data, metadata, or data delivery systems.
It is maintained by the Southern Regional Climate Center and NCDC is a participant to the extent of addressing data it holds and systems it maintains.
33 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Errors vs. CorrectionsErrors vs. Corrections
• Datzilla intended to correct Datzilla intended to correct errorserrors• CorrectionsCorrections to observations are to observations are not errorsnot errors..• For changes to recent observations (For changes to recent observations (commoncommon))
– William Angel (COOP / ASOS)William Angel (COOP / ASOS)– Blake Lasher (LCD ASOS)Blake Lasher (LCD ASOS)– Send amended form to NCDC via normal channels.Send amended form to NCDC via normal channels.
• Changes to historical observations (Changes to historical observations (uncommonuncommon))– Amended formsAmended forms– Datzilla (limited to changing estimates of invalid obs)Datzilla (limited to changing estimates of invalid obs)– Must have a compelling reason to change a historical obs.Must have a compelling reason to change a historical obs.
• Corrections due to equipment problems may also require Corrections due to equipment problems may also require a metadata rendition.a metadata rendition.
44 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
When NCDC receives a Datzilla When NCDC receives a Datzilla ticket …ticket …
• Datzilla gatekeeperDatzilla gatekeeper
• Initial determinationsInitial determinations
• ReassignmentReassignment
• InvestigationInvestigation
• Course of actionCourse of action
• ResolutionResolution
• ClosureClosure Your friendly Datzilla Gatekeeper
55 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Some possible reasons for a Some possible reasons for a Datzilla ticket to NCDCDatzilla ticket to NCDC
• Data issue (usually)Data issue (usually)– That TMAX of 74 should be a 47!That TMAX of 74 should be a 47!– Your #$&!@* QC clobbered my data!Your #$&!@* QC clobbered my data!
• System issueSystem issue– CDO inventory doesn’t match the data I got!CDO inventory doesn’t match the data I got!
• Metadata issueMetadata issue– This station’s COOP number is wrong!This station’s COOP number is wrong!
66 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Good tickets and BadGood tickets and Bad
• Pre-report investigation (critical)Pre-report investigation (critical)– Is the error really an error?Is the error really an error?– Is it the right station?Is it the right station?– Is it reproducible (browsers, systems)Is it reproducible (browsers, systems)
• Do you have the right system?Do you have the right system?– ACIS, ThreadEx, NCDC data may be differentACIS, ThreadEx, NCDC data may be different
• Have you proposed a solution?Have you proposed a solution?
77 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
A good A good ticketticket
• Clear titleClear title• Good Good
description of description of problemproblem
• ID of stationID of station• Attached formAttached form• Proposed Proposed
solution.solution.
88 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
A bad ticketA bad ticket
• Vague titleVague title
• Vague description of problemVague description of problem
• No ID of specific station, element, dateNo ID of specific station, element, date
• No supporting materialNo supporting material
• No suggested fixNo suggested fix
• HunchesHunches
• Data “corrections”Data “corrections”
99 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Investigating the errorInvestigating the error• xmACIS is a display tool for NOAA dataxmACIS is a display tool for NOAA data
– Interprets data flags from NCDC data setsInterprets data flags from NCDC data sets– Prioritizes simultaneous values from different data setsPrioritizes simultaneous values from different data sets– Displays “M” for invalid as well as missing dataDisplays “M” for invalid as well as missing data– May thread data from several stationsMay thread data from several stations
• Primary, Secondary, Tertiary forms:Primary, Secondary, Tertiary forms:– 1001, B-91, 10-A, …1001, B-91, 10-A, …– 1014, LCD, CD, CRB1014, LCD, CD, CRB– Media articles, eye-witness accounts, proxy dataMedia articles, eye-witness accounts, proxy data
• Paper, WSSRD, IPS …Paper, WSSRD, IPS …
1010 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
My ticket’s been open forever!My ticket’s been open forever!
• Some are easy to fixSome are easy to fix– Small-batch keying errorsSmall-batch keying errors– A few missing valuesA few missing values
• Some are notSome are not– May be part of a larger issueMay be part of a larger issue– May exceed NCDC resourcesMay exceed NCDC resources– May not have a straightforward fixMay not have a straightforward fix– May involve parties/systems outside NCDCMay involve parties/systems outside NCDC
1111 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Your #$&!@* QC clobbered my Your #$&!@* QC clobbered my data!data!
• You are right to alert us.You are right to alert us.
• However, before you do …However, before you do …– Think about Think about whywhy our QC would flag that data. our QC would flag that data.– Most often, the problem is not our QCMost often, the problem is not our QC
• Observer date shiftingObserver date shifting
• Observing at time different than expected.Observing at time different than expected.
– Known climate anomalies?Known climate anomalies?– Unreported temporary anomalies?Unreported temporary anomalies?
1212 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Should that 850° F TMAX really be invalidated?
Fuel for the FireFuel for the Fire
1313 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
One NOAAOne NOAA
• We’re all interested in having the best We’re all interested in having the best NOAA data possible.NOAA data possible.
• Better data means more accurate Better data means more accurate forecasts and research.forecasts and research.
• NCDC has limits on what it can do to NCDC has limits on what it can do to data.data.
• We will address and work to fix all We will address and work to fix all legitimate errors.legitimate errors.
1414 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Any questions?
Once again, your friendly (no, not that friendly) NCDC Datzilla Gatekeeper!
828.271.4223
NCDC Halloween, 2006