The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin,

The Unified Access FrameworkThe Unified Access Frameworkfor Gridded Datafor Gridded Data

… … the 1the 1stst year focus of NOAA’s Global year focus of NOAA’s Global Earth Observation Integrated Data Earth Observation Integrated Data

EnvironmentEnvironment (GEO-IDE)(GEO-IDE)

Steve Hankin, NOAA/PMELSteve Hankin, NOAA/PMELand the NOAA UAF teamand the NOAA UAF team

NOAA UAF-gridNOAA UAF-grid 22

Enterprise-wide integration of data Enterprise-wide integration of data is a very hard problem! is a very hard problem!


Heterogeneous requirementsHeterogeneous requirements

Data Providers with different outlooksData Providers with different outlooks(different "corporate cultures”)(different "corporate cultures”)

Data Users with diverse needs and skillsData Users with diverse needs and skills

Corporate cultures and people’s habits are Corporate cultures and people’s habits are notoriously difficult to changenotoriously difficult to change

… … and by the way, getting funds for integrationand by the way, getting funds for integrationis often a hard sellis often a hard sell


Integrating NOAA’s dataIntegrating NOAA’s datais such a challengeis such a challenge

weather forecast weather forecast (time critical)(time critical)

fisheries management fisheries management (regulatory aspects)(regulatory aspects)

nautical chartingnautical chartingclimate, ocean, atmosphere monitoringclimate, ocean, atmosphere monitoring… …

modeling & observationsmodeling & observations

research & operationsresearch & operations


The accepted approach:The accepted approach:build a “system of systems”build a “system of systems”

Connect existing data systems using aConnect existing data systems using aService Oriented ArchitectureService Oriented Architecture(“wrap” existing systems with loosely coupled, (“wrap” existing systems with loosely coupled, standardized services)standardized services)


Global Earth ObservationGlobal Earth Observation

Integrated Data EnvironmentIntegrated Data Environment

GEO-IDE Con-ops GEO-IDE Con-ops outlines such a planoutlines such a plan

2005-06, 66 pages2005-06, 66 pages


Seed fundingSeed funding ٨٨ available this yearavailable this year

How do you approach a very big problem with a How do you approach a very big problem with a very small amount of money?very small amount of money?

finallyfinally


1.1. Define requirementsDefine requirements

2.2. Create enterprise architecture documentsCreate enterprise architecture documents

3.3. Assemble volunteer teams to implement Assemble volunteer teams to implement

NOT !NOT !


An alternative approachAn alternative approach

Don't Solve ProblemsDon't Solve Problems-- Copy Success -- Copy Success

""Switch: How to Change Things When Change Is HardSwitch: How to Change Things When Change Is Hard", ", Chip and Dan Heath (psychologists)Chip and Dan Heath (psychologists)

http://www.fastcompany.com/magazine/142/switch-how-to-change-things-when-change-is-hard.html

Feb. '10 - Ocean SciencesFeb. '10 - Ocean Sciences NOAA UAF-gridNOAA UAF-grid 1010

Projects: (too many to name)Projects: (too many to name)

DataDataformats:formats:

netCDF GRIB HDF

ServiceServicestack:stack:

netCDF-CF-DAP-THREDDS-WMS

Applications:Applications: Matlab ArcGIS Ferret

GrADS Google Earth IDV LAS ERDDAP …

Users: (too many to name)Users: (too many to name)

…


UAFUAF(*)(*)-grid-grid(initial focus: gridded data)(initial focus: gridded data)

– netCDF netCDF – CF CF – OPeNDAP OPeNDAP – THREDDS THREDDS – WMS (& WCS) WMS (& WCS)

(*) Unified Access (*) Unified Access FrameworkFramework

---------- an abstract data an abstract data modelmodel

---------- metadata conventionsmetadata conventions

---- network transportnetwork transport

---- network topologynetwork topology

---- renderingrendering


Who is using this approach?Who is using this approach?

Climate, ocean, atmosphere modelersClimate, ocean, atmosphere modelers– IPCC, GFDL, NCAR, …IPCC, GFDL, NCAR, …

Satellite programsSatellite programs– GHRSST, PathFinder, CoastWatch, …GHRSST, PathFinder, CoastWatch, …

NCEP weather and ocean forecastsNCEP weather and ocean forecasts(GRIB via NOMADS)(GRIB via NOMADS)

Coastal (“HF”) radarCoastal (“HF”) radar

A growing list of observations programsA growing list of observations programs– Argo, OceanSites, SAMOS&GOSUD (soon), …Argo, OceanSites, SAMOS&GOSUD (soon), …


Some rough spotsSome rough spots

need better compliance with CF conventionsneed better compliance with CF conventions

"files" need to be aggregated into logical "files" need to be aggregated into logical datasetsdatasets

need richer discovery-level metadataneed richer discovery-level metadata


NcMLNcMLa tool for fixing compatibility problemsa tool for fixing compatibility problems

e.g. adding a CF “standard_name” attribute to GRIB datae.g. adding a CF “standard_name” attribute to GRIB data

<variable name="vorticity"><variable name="vorticity"> <attribute <attribute name="standard_name" name="standard_name" value="atmosphere_absolute_vorticity“ value="atmosphere_absolute_vorticity“ /> /></variable></variable>

The file, itself, is untouched. But access to the file through the The file, itself, is untouched. But access to the file through the services is made to conform to standards.services is made to conform to standards.


more NcMLmore NcML

e.g. aggregating 3 years of data into 1e.g. aggregating 3 years of data into 1

<aggregation type="joinExisting“<aggregation type="joinExisting“ dimName="TimeAxis"> dimName="TimeAxis"> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/>

</aggregation></aggregation>

The files are untouched. But the services see a single logical The files are untouched. But the services see a single logical file.file.


How users will find UAF datasetsHow users will find UAF datasets

Less mature …Less mature …

UAF web crawler extracts THREDDS metadataUAF web crawler extracts THREDDS metadata– Ingests into a dbaseIngests into a dbase– other metadata also ingested as available other metadata also ingested as available

ISO 19115 records generated ISO 19115 records generated Datasets and services registered Datasets and services registered – GCMD, Geospatial One-Stop, GEOSS, data.gov GCMD, Geospatial One-Stop, GEOSS, data.gov

Much metadata work will remain beyond year 1.Much metadata work will remain beyond year 1.


The emerging UAFThe emerging UAF

The UAF THREDDS catalog:The UAF THREDDS catalog:

~400 “clean” datasets~400 “clean” datasets.. growing as the team aggregates .. growing as the team aggregates and configures using ncMLand configures using ncML– team members become coachesteam members become coaches

lasting solutions must be implemented lasting solutions must be implemented at the data provider sitesat the data provider sites

The UAF TeamThe UAF Team– 8-10 active participants8-10 active participants– self-selectedself-selected

5 months into the project …5 months into the project …


The emerging UAFThe emerging UAF

Users access data through Users access data through their client applicationstheir client applications– Without downloading filesWithout downloading files

Configuration of clients is Configuration of clients is underwayunderway

Some examples …Some examples …Take differences

Scripts

Live Access Server Live Access Server (LAS)(LAS)


Model 1: UMASS-ECOMModel 1: UMASS-ECOM Model 2: UMAINE-POMModel 2: UMAINE-POM Data: SST 2008-Sep-08 07:32Data: SST 2008-Sep-08 07:32

Desktop access in MatlabDesktop access in Matlab


Desktop access in FerretDesktop access in Ferret


Desktop access in GrADSDesktop access in GrADS


Desktop access in IDVDesktop access in IDV


Access in ArcGISAccess in ArcGISusing the Environmental Data using the Environmental Data

Connector (EDC)Connector (EDC)


Access in Google EarthAccess in Google Earth


WMS provides WMS provides access to many access to many GIS applicationsGIS applications


Browser access using ERDDAP. Browser access using ERDDAP. And from there …And from there …


.mat.mat

REST URL access to data subsetsREST URL access to data subsetsin several simple formatsin several simple formats

(accessible through home-grown scripting of many types)(accessible through home-grown scripting of many types)

.kml

.mat

.nc


Next year’s target: observational data

Collections of time series & profiles arealready on-line

UAF home page … coming soon


““No standard should be approved without having No standard should be approved without having been used to implement a few projects of realistic been used to implement a few projects of realistic complexity.”complexity.”

The Rise and Fall of CORBA (Henning, 2006)

The UAF strategy buildson historical lessons


Thank youThank you

Documents

The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin,