Data Discovery and Access to The International Surface Pressure Databank (ISPD)

1

Data Discovery and Access to The International Surface Pressure

Databank (ISPD)

Thomas Cram Gilbert P. Compo*Doug Schuster Chesley McColl*Steven Worley

National Center for Atmospheric Research, Boulder, CO*NOAA/CIRES, Boulder, CO

AMS 2013 Annual Meeting: 29EIPT/25CVC J1.2

2

Research Data Archive (RDA) at NCARrda.ucar.edu

1. 600+ distinct datasets for climate and weather research

2. Collections: ocean & atmosphere observations, analyses, reanalyses, operational NWP outputs

3. Free and open accesshttp://rda.ucar.edu


3

ISPD Overview• World’s largest collection of surface & sea level pressure observations

Land station Marine observations (ICOADS) Tropical cyclone best track

(IBTrACS)• Period (version 2): 1768 – 2010• Volume: 465 Gbyte• Available since Aug 2010


4

ISPD Overview• 60+ Contributors

o Atmospheric Circulation Reconstructions over the Earth (ACRE)o Australian Bureau of Meteorologyo British Antarctic Surveyo Cook Islands Meteorological Serviceo Danish Meteorological Instituteo Deutscher Wetterdienst (DWD; German Weather Service)o European and North Atlantic Daily to Multidecadal Climate Variability

(EMULATE)o ETH Zurich, Switzerlando GCOS/WCRP Working Group on Observational Data Sets for

Reanalysiso MANY MANY MORE….

• Assembled by NOAA/ESRL, CIRES (Univ. of Colorado), & NOAA/NCDC


5

ISPD Sample Annual Station Distribution

1850 * Land stations only* No marine stations


6

18501900

* Land stations only* No marine stations



7

18501900

1950




8

18501900

19502000



9

ISPD Observations/Year

Figure courtesy Chesley McColl, NOAA/ESRL

2010: 53 Million

~ 1.5 Billion total observations

10

Data Access: Problem Background• Large computational/storage resources

needed– Store data– Extract desired data from large grids/files– Convert data to desirable format(s)

Scientific data centers have these resources

Individual researchers generally don’t


11

• Goals– Make data more accessible and easier to use for

individual researchers• Reasonable access volumes• Desired data formats• User defined parameters/grids

• Researchers stay focused on research

Data Access: Problem Background


12

ISPD Data Access Services

• Powerful computing resources @ NCAR

• Large disk storage (~ 0.5 PB)• Rich and detailed metadata• Direct file download via web• Customized data sub-setting• HDF-5 to ASCII software tools

(NetCDF forthcoming)

13

ISPD Metadata Features• Both group- and file-level metadata• Drive interfaces for file grouping and sub-

setting tools• Support efficient back-end processing• Improve scalability• Provide “quick look” at data samples


14

ISPD Metadata Interface Example


15AMS 2013 Annual Meeting: 29EIPT/25CVC J1.2

16AGU 2012 Fall Meeting: IN44A-05

17AMS 2013 Annual Meeting: 29EIPT/25CVC J1.2

18

Data Access: ISPD Subset Interface


19

ISPD Data Access ServicesTemporal range sub-

setting (daily)

Spatial sub-setting Lat/Lon region Individual station ID

20


Data sub-setting options (cont.)• Observation type

Land station Marine obs Radiosonde Dropsonde TC best track

21


• Subsetting processed in delayed mode• E-mail notification• Download via server-provided scripts

(wget)

22

ISPD 2012 Subset metrics

Data accessed: ~ 6.5 TB

Data served: ~ 46 GB


23

Summary & Future Directions

• RDA – Supply “User Friendly” Data Parameter & spatial sub-setting Metadata discovery Format conversion Improved and additional services

• NWSC-Cheyenne opening – more computing power


24

• DOI assignment

• Geoscience Data Journal article

• ISPD v3 (1755-2010) Spring 2013

ISPD Forthcoming


25

rda.ucar.edu/[email protected]


Documents

Data Discovery and Access to The International Surface Pressure Databank (ISPD)