Upload
jonathan-nichols
View
220
Download
0
Embed Size (px)
DESCRIPTION
Data ServicesData Services
Citation preview
GES DISC ServicesPush Harder?Be Careful?Change Direction?What about adding ______?
Discovery Services Mirador
Development scaled back to sustaining engineering level
External Search (in Test mode TS1) Technically successful, but... Usability-challenged
Start and stop date/time Total number of hits Uniform sort order Duplicates
Usability: Simplicity vs. Features (esp. Services) Mirador Usability Sounding Board?
mail list for queries on usability quandaries
Data Services
Number of Users* - March 2011
NativeGiovanniOGCnetCDFSubsettingOPeNDAPDQSS
*OK, not really. It’s the number of distinct IP addresses
Number of Users*: Sep 2010 – Apr 2011
201
201
201
201
201
201
201
201
0100020003000400050006000700080009000
DQSSGiovanninetCDFSubsettingOPeNDAPOGCNative
Data Quality Screening Service
The quality of AIRS data varies considerably
AIRS Parameter Best (%)
Good (%)
Do Not Use (%)
Total Precipitable Water
38 38 24
Carbon Monoxide 64 7 29Surface Temperature
5 44 51Version 5 Level 2 Standard Retrieval Statistics
Quality Schemes can be complicated
Hurricane Ike, viewed by the Atmospheric Infrared Sounder (AIRS)
PBest : Maximum pressure for which
quality value is “Best” in temperature profiles
Air Temperatureat 300 mbar
Current user scenarios...
Nominal scenario Search for and download data Locate documentation on handling quality Read & understand documentation on
quality Write custom routine to filter out bad pixels
Equally likely scenario (especially in user communities not familiar with satellite data) Search for and download data Assume that quality has a negligible effect
Repeat for
each user
The effect of bad qualitydata is often not
negligible
Total Column Precipitable
WaterQuality
Best Good Do Not Usekg/m2
Hurricane Ike, 9/10/2008
DQSS replaces bad-quality pixels with fill values
Mask based on user criteria(Quality level
< 2)
Good quality data pixels
retained
Output file has the same format and structure as the input file (except for extra mask and original_data fields)
Original data array(Total column precipitable water)
DQSS Status + Plans
Operational for AIRS L2 Standard Retrieval Nearly operational for MODIS Water Vapor Next: MODIS Aerosols, MLS Water Vapor Next: ??? Also, OPeNDAP Gateway nearly reader to
front-end DQSS Allow OPeNDAP access to DQSS-served data.
OPeNDAP*
Remote access to data: no need to download Access at fine granularity
Variable Array regions Stride
Present HDF data as netCDF/CF Enhances Tool Usability
Reformatting: ASCII, netCDF
*OPeNDAP = OpenSource Project for a Network Data Access Protocol
Who Uses OPeNDAP?
Industrial-strength scripters looking for subsets
Thick client users GrADS, Panoply, IDV, McIDAS-V, Ferret
Internal Systems Giovanni MapServer Simple Subset Wizard
OPeNDAP Demo
OGC* Standards - WMS
Web Map Service (WMS) URL request: returns map image Implemented with open-source MapServer
Giovanni also supports WMS Consumers:
AIRS NRT page Google Earth GIS programs IDV Giovanni
*OGC = Open Geospatial Consortium
OGC - WCS
Returns “coverages”: data variables in NetCDF/CF1
Used by other systems DataFed Giovanni Atmospheric Composition Portal Simple Subset Wizard
Subsetting
Semi-custom tools for some products Reuse HSE libraries from UAH Reuse Lats4D from A. DaSilva
Usually HDF in -> HDF out Implemented as REST* URLs
Subsetting at time of download Subsets are implemented as user requests come in Areas where we should proactively develop
subsetters?
~100 Subsettable Datasets
AIRS Radiances (channel), L2 Retrievals (variable), L3 (spatial+variable via SSW)
MLS L2 (spatial+variable) TOMS L3, OMI L2-L3 (spatial+variable), OMI L2 TRMM L3 (spatial+variable) Models (spatial+variable) Did we miss any (that shouldn’t be missed)?
Should all SSW subsets be offered in Mirador?
Format Conversion
Custom code for some L3 and L2 datasets HDF -> netCDF/CF Improves usability in tools Moving toward external tools where possible
OPeNDAP Lats4d: based on GrADS
Simple Subset Wizard
Desired: “Just give me the data from time 1 to time 2 for this spatial box”.
Current: “search for granules, view granules, select granules, select subset option, re-enter spatial box...”
ESDIS-funded technology infusion effort DEMO
Giovanni Evolution
G3 Evolution to Agile Giovanni (G4)
Factors driving evolution G3 architecture was never completed
No workflow engine Cost of adding significant features is too high
Architecture is too brittle
Key G4 Goals
Reduce cost and time to add new features Improve performance over G3 Support external maintenance of external
data
Evolution Plan
Implement new projects in Agile Giovanni (G4) Aerostat ACCESS project
Point data in database, bias corrections Year of Tropical Convection (YOTC)
Level 2 data Community-based Giovanni
Externally maintained portals and data
Implement G4 features to meet existing G3 functionality
Migrate G3 instances to G4 portals
Roads Not Taken
Giovanni 3 enhancements
ISO 19115 Metadata Document
architecture Mirador features and
usability revamp Persistent locators Unique identifiers
Not Giovanni Evolution DQSS Atmospheric Composition
Portal Simple Subset Wizard Community-based
Initiatives Mirador External Search Expanding data services
Taken
Backup Slides
Agile Giovanni Architectural Features
Model-view-controller Semantic Web underpinnings Variable-centric, not dataset-centric Code reuse: Kepler, YUI, JCache, MapServer