Embed Size (px)
Citation preview
Virtual Quality Screening Service (VQSS): Improving the application of quality informationNASA funded Advancing Collaborative Connections for Earth System Science
(ACCESS) Project
Ed Armstrong, Thomas Huang, Zhangfan Xing, Christian Alacron, Toshio Chin, NASA JPL
Siri Jodha Khalsa, NSIDC
Optimal use of satellite-based earth science data records requires access to and understanding of the data quality information contained in those records. This can be a complex and time-consuming process, with metadata attributes, bit flags, ancillary variables all needed, possibly in combination, to ensure that the data meets scientific requirements. For example, quality screening of Level 3 data from the upcoming Soil Moisture Active Passive (SMAP) instrument can involve up to 26 unique bit states or conditions a user can filter for. For GHRSST L2P granules a minimum of 10 variables can be used to screen for SST data on a pixel-by-pixel basis.
VQSS Introduction and Motivation
The Virtualized Quality Screening Service (VQSS), a recently funded 2014 NASA ACCESS project, aims to address these issues and concerns by developing an infrastructure that will allow users to view and apply the quality information in SMAP and other products from the Group for High Resolution Sea Surface Temperature (GHRSST) Project. It leverages proven NASA components for data extraction, subsetting by value, and visualization using granule-based quality information.
Builds on existing PO.DAAC web services for data (and metadata) discovery, subsetting and extraction, and visualization. See PO.DAAC web services homepage http://podaac.jpl.nasa.gov/ws/
VQSS.…
Implement web services and related infrastructure for quality screening of GHRSST and SMAP L2/L3 data. Allow users to explore meaning and effect of quality variables. Share and store URLs to explicitly extract and screen geophysical data from granules.
How? Expose data granules from GHRSST/SMAP mission through webification Extend the PO.DAAC web services, also known as the EDGE as an external data service
for granule searching Provide a portal for the public exposure of VirtualQSS that will allows users to:
Search for SMAP granules using spatial-temporal constraints Review quality,error estimates and others ancillary variables and information specific to a
unique data type exposed through a semantic layer or lexicon Apply quality flag filtering based on exact user specifications to granules Subset and return filtered results as single granules and/or aggregates in preferred formats
such as netCDF, HDF, JSON, and CSV Store and share filtering queries with other scientists and the community
Objectives of VirtualQSS
A minimum 10 variables in a GHRSST granule could potentially be used to filter SST observations. More if additional experimental variables are provided.
quality_level (scalar flags) l2p_flags (bit flags for various conditions) wind (physical variable) sses_bias (error statistic: bias in degC) sses_sd (error statistic: standard deviation) sea_ice_fraction (physical variable) aerosol (physical variable) dt_analysis (anomaly SST ) satellite_zenith_angle (instrument geometry) solar_irradiance (physical variable) …other experimental variables like diurnal warming SST/CHL_A/K490/Brightness Temperature
depending on specific dataset variables
GHRSST “quality” variables for L2P data
Examples: short l2p_flags(time, nj, ni);
l2p_flags:flag_meanings = "microwave land ice lake river reserved_for_future_use no_retrieval N2_retrieval N3R_retrieval N3_retrieval D2_retrieval D3_retrieval cloud sun_glint cosmetic_fill
l2p_flags:flag_masks = 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s, 1024s, 2048s, 4096s, 8192s, 16384s, 32768s ;
byte quality_level (time, nj, ni); quality_level:flag_meanings = "no_data bad_data worst_quality low_quality acceptable_quality best_quality"; quality_level:flag_values = 0b, 1b, 2b, 3b, 4b, 5b;
CF metadata for describing quality
Example L3 flags for SMAP
Also contains multiple variables for quality screening. Mostly in the form of bit flags. 26 unique bit states or conditions a user can filter for L3 data.
Webification (w10n) is an enabling technology that simplifies use of large and complex science data, such as ones archived at PO.DAAC, using HTTP/HTTPS protocols with URLs comprised of well-defined parameters. Similar to OPeNDAP.
W10n abstracts an arbitrary data store as a tree, in which two types of entities exist: node and leaf.
Direct access to inner components of the node/leaf is via HTTP requests from either a web browser, script or similar client. Results of W10n calls return specified measurement arrays or metadata elements via subset by array value (v.s. subset by array index), according to supported output formats (JSON, HTML, netCDF) as specified in the URL request.
Webification
More W10n SST query examples
Example of webification requests for a MODIS Terra sea surface temperature granule.
• http://host:port/path/2013/123/20130503-MODIS_T-JPL-L2P-T2013123065500.L2_LAC_GHRSST_N- v01.nc.bz2/sea_surface_temperature[-130<lon<- 120,35<lat<45]?output=json
• http://host:port/path/2013/123/20130503-MODIS_T-JPL-L2P-T2013123065500.L2_LAC_GHRSST_N- v01.nc.bz2/sea_surface_temperature[quality_flag>=4]?output=nc
• http://host:port/path/2013/123/20130503-MODIS_T-JPL-L2P-T2013123065500.L2_LAC_GHRSST_N- v01.nc.bz2/sea_surface_temperature[quality_flag>=4,wind_speed>6,-130<lon<- 120,35<lat<45]?output=nc.4
• Final request is for spatial subsetting with a specific quality flag levels chosen and a wind threshold of 6 m/s selected. Output is subsetted netCDF4.
VQSS Architecture
Architecture of VQSS system. Webification is linked directly to the JPL PO.DAAC to serve its granules. Other data sources such as SMAP from NSIDC are served via “proxy.” VQSS will leverage and extend a granule and metadata discovery service at the PO.DAAC called EDGE to expose additional quality attributes and metadata for SMAP. A user interface will be developed to expose all these services to the science community.
Allow users to search for relevant granules (e.g., GHRSST granules from a specific sensor, or soil moisture datasets from SMAP)
Expose quality information for each data type Provide a mechanism to abstract the quality screening/subset request Federate and balance w10n screening requests Access returned results from the w10n servers in preferred output
formats Provide a method to save queries for future reference or sharing Visualize results
Portal
OceanXtremes: Oceanographic Data-Intensive Anomaly Detection and Analytics Portal PI: Thomas Huang/JPL, Co-I: Brian Wilson/JPL, George Chang/JPL, Ed Armstrong/JPL, and Toshio
Chin/JPL
Mining and Utilizing Earth Science Dataset Metadata, Usage Metrics, and User Feedback to Improve Dataset Relevancy PI: Chaowei (Phil) Yang/GMU, Co-I: Ed Armstrong/JPL, Thomas Huang/JPL, and David Moroni/JPL
A Service to Match Satellite and In-situ Marine Observations to Support Platform Intercomparisons, Cross-calibration, Validation, and Quality Control PI: Shawn Smith/COAPS, Co-I: Thomas Huang/JPL, Vardis Tsontos/JPL, Ben Holt/JPL, Mark
Bourassa/COAPS, and Steve Worley/NCAR
2014 NASA Advanced Information Systems Technology (AIST) proposals
Working with large volume satellite data (Big Data) presents complex challenges with accessing relevant research information. This issue is only getting worse.
New tools and services are needed to discover, subset, extract, and visualize smaller volumes
The Virtual Quality Screening Service will offer a key contribution in this web services paradigm by providing the ability to apply any quality or ancillary variables as filters to the physical variable of interest
To be deployed on GHRSST and SMAP L2/L3/L4 granules in 2015
Summary
Goal: Expose quality information in a user friendly fashion OCSI search results return quality parameters Build a lexicon
For example, user floats over variable or metadata attribute detailed information will be exposed
Menu choices of quality variables or quality levels Ontology development
Formal dataset specific ontology linking variables with quality concepts including CF and ISO
May make it easier to extend to other datasets beyond GHRSST/SMAP
Quality information exposure
PO.DAAC SST data GHRSST:
L2P MODIS_A, MODIS_T, VIIRS, AVHRR, AMSR-E, AMSR2 L3C AMSR2, AVHRR L4 Global
NSIDC SMAP data L2/3 Active, Passive and A/P datasets
Future Integration plans
Year 1: Implement local w10n server for all GHRSST datasets residing in the PO.DAAC Develop quality parameter lexicon adaptation for GHRSST data/metadata model Enhance OCSI interface to include quality parameters and delegate to use ECHO for SMAP
test data Create W10n proxy implementation of NSDIC OPeNDAP servers.
Year 2: Implement services in beta version of web portal. Extend quality lexicon, granule searching, data filtering to support operational SMAP data. Test and finalize portal. Deploy portal and integrate with PO.DAAC Labs. Package software. Engage community for
testing. Package webification and other software for open source
Milestones
