Improving an Air Quality Decision Support System through the Integration of Satellite Data

with Ground-based, Modeled, and Emissions Data

NASA ROSES 2007: Decision Support through Earth Science Research Results

Project Update 1/7/2010: Cooperative Institute for Research in the Atmosphere

Project overview

NASA ROSES proposal to enhance VIEWS/TSS with NASA satellite data

•Based on the NASA ROSES omnibus solicitation of 2007

•We submitted the original proposal in 2007 or 2008 (?)

•Proposal was awarded in 2008

•Work began in 2009 after NASA funding was received

•Uma Shankar (UNC) and Shawn McClure (CIRA/CSU) are the co-PIs

•Many collaborators, including Dr. Bret Schichtel (NPS) and Tom Moore (WRAP) at CIRA

•Two people from CIRA working part time on the project

•Multiple people from UNC working part time on the project

•Period of performance ends in May 2011

Project goals

Overall Project Goal

To enhance and add value to a currently operational air quality decision support system (VIEWS/TSS) by integrating and utilizing satellite data from NASA satellites Aura, Terra, Aqua, and CALIPSO.

Specific Project Goals

• Develop routine capture, analysis, and processing algorithms with high temporal and spatial resolution to provide land use/land cover data as inputs to emissions and air quality modeling

• Achieve more complete temporal and spatial resolution of activity data and emission rates from natural and anthropogenic emission sources in remote areas and from individual sources and source clusters

• Obtain multiple-dimensional vertical profiles and column measurements of pollutants to improve model inputs and provide evaluation data for gridded chemistry-transport models such as CMAQ

• Develop advanced analysis tools to better understand the relevant atmospheric processes and their representation in the CTMs

• Visualize and analyze satellite data in combination with existing monitoring, emissions, and modeling data within a unified decision support platform

Introduction to VIEWS/TSS

The Visibility Information Exchange Web System is a database system and set of online tools originally designed to support the Regional Haze Rule enacted by the EPA to reduce regional haze in national parks and wilderness areas.

What is VIEWS?

• Serve as the primary online source for IMPROVE aerosol and RHR datasets

• Provide easy online access to a wide variety of ground-based air quality data

• Provide online tools for exploring, analyzing, and downloading this data

• Maintain a catalog of relevant air quality-related resources

• Facilitate the research and understanding of global air quality issues

What its primary goals?

An extension to VIEWS developed by the Western Regional Air Partnership (WRAP) to provide a single portal to technical data and analytical results prepared by WRAP Forums and Workgroups, and to serve as a decision support system for aiding the development of state and tribal implementation plans.

What is the TSS?

http://views.cira.colostate.edu

http://vista.cira.colostate.edu/tss

Simplified, CIRA-specific project goal

Make it easier to…

• find

• visualize

• query

• download

…satellite and modeled data in conjunction with ground-based data

Fulfilling our goal: Getting the data

Fulfilling our goal: Importing and integrating the data

Fulfilling our goal: Enhancing our system to use the data

Leveraging the VIEWS Architecture

Project Overview: What we’ve done on the back end

• Imported a large collection of satellite and modeled air quality data

• Implemented an extensive software library for querying, filtering, managing, and reformatting HDF and netCDF files, both programmatically and remotely

• Acquired expertise in manipulating, managing, and analyzing satellite data for use in air quality applications

• Designed an experimental spatial relational database schema for managing gridded satellite and modeled data

• Developed an automated suite of utilities to dynamically generate images from HDF and netCDF files

• Designed a folder and file naming convention for facilitating the programmatic management, exploration, and parsing of data files

• Implemented a suite of temporal aggregation utilities that can operate on native satellite and modeled data files

• Designed strategies and methods for performing meaningful spatial interpolations of gridded and point-based data

• Developed uniform KML/KMZ schemas for representing point data

Project Overview: What we’ve done on the front end

• Designed a 2D mapping tool using Google Maps that can display satellite, modeled, and ground based datasets

• Developed an interface for querying (“drilling down” through) a set of spatial layers to retrieve and display data values at a selected point on a map

• Developed a timeline tool for visualizing the temporal availability and coverage of our inventory of satellite, modeled, and ground based datasets

• Developed a general purpose file “explorer” (similar to Windows Explorer) for browsing and retrieving documents and data files

• Developed a tool for browsing, selecting, and animating the large collection of static images generated from the source data files

• Implemented a very loosely-coupled integration of UNC Tools (AMET, VPM) with the VIEWS website

• Designed access points and client-server communication mechanisms to the new tools from existing pages on VIEWS

• Developed a suite of visual, interactive test harnesses for exercising and refining the various software components developed for this project.

Project Overview: What we haven’t done

• Achieved a more thorough and pervasive integration of the new tools with the existing tools and web pages

• Developed end-to-end, real world case studies that “mash up” individual tools, components, and services into polished end user analyses and reports

• Realized a more comprehensive integration with remote data systems such as RSIG and Giovanni

• Integrated the satellite and modeled data more thoroughly and transparently into the spatial/relational database, and utilize this integration from the tools. (NOTE: This “lack” of a fuller RDBMS integration was partially due to both intentional design decisions and also unexpected complexities.)

• Achieved a tighter, more seamless integration of the UNC tools (AMET, VPM) with VIEWS website and database

• Made as much progress as hoped/expected on the visualization and analysis tools

Challenges and issues encountered

• Key personnel resources were lost mid-project (details and ramifications?)

– A key satellite scientist had to be released due to unexpected budgetary constraints (CIRA)

– Two lead developers took different positions during intensive development phases (UNC)

• An unexpectedly high degree of heterogeneity was encountered in satellite data formats and conventions

• The quality and completeness of available satellite data documentation was very non-uniform and often difficult to find and understand

• Many satellite measurements were found to have limited direct relevance to meaningful air quality studies

• Low-level data management of satellite data was found to be a more complicated and “one-off” exercise than hoped

• Relational database capabilities were found to be lacking with regard to handling densely-gridded data

• Data import, management, and post-processing, as well as software development, was more complicated and took longer than hoped

Import: Organize the source data files on a file server

Import: Apply a strict file and folder naming convention

• Aura_OMI_L3• ColumnAmountO3• CloudFraction• World• netCDF• 20070101• nc

← <Satellite>_<Instrument>_<Processing level>← <Parameter group>← <Parameter>← <Spatial extent>← <File format>← <Temporal extent>← <File extension>

Individual components of the folder path and file name:

Resulting file name:

Aura_OMI_L3!ColumnAmountO3!CloudFraction!World!netCDF!20070101.nc

Import: Understand the structure and contents of the data

Import: Understand the context of the data (i.e. metadata)

• Sourceo Satellite platformo Atmospheric modelo Data provider

• Collection Method(s)

• Processing Method(s)

• Value interpretationo Concentrationo Integrated column amounto Radiance measureo Etc…

• Units, precision, significant digits

Import: Extract relevant subsets of the data

Import: Cross reference parameters across datasets

Derived data products

• Uniformly formatted netCDF files– Optimized for long term storage– More easily consumed by end users– Facilitates interoperability and standardization– Better enables derived product generation

• Static graphical images– Can be used by a wide variety of visualization tools– Transparency of *.PNG images is ideal for Google and Bing Maps– Enables quick browsing and identification of events– Enables client-side animations

• KML and KMZ files– Makes it possible to store feature and topology data– Can be used by current and popular visualization tools– Enables direct data and metadata query– Facilitates sharing and interoperability

Accessing project data and tools from the main menu

File Explorer tool

Image Browser and Animation Tool

Dataset Timeline and Parameter Inventory tool

Map Viewer: OMI Level 3 Ozone Column

Map Viewer: OMI Level 3 Radiative Cloud Fraction

Map Viewer: IMPROVE Ammonium Sulfate

Map Viewer: Three layers combined

Viewing data products in Google Earth

Test harnesses and alpha tools

Exceptional event metadata (such as fires) can be stored and dynamically associated with data, photos, and imagery at run time in order to provide quick cross-referencing and discovery of related data. Thus, satellite and modeled data can be quickly associated with aerosol events once these linkages are complete.

“Bisquit” fire impacting Crater Lake in 2002

Roadmap: Linking data to relevant photos and imagery

Roadmap: Speciated contribution to air quality case studies

Fractional SO4 AOD

Fractional NO3 AOD

Fractional NH4 AOD

Fractional OC AOD

Fractional BC AOD

Fractional A25 AOD

Fractional CM AOD

August 2001

Fract. AOD

Sulfate contributes ~ 40%

Roadmap: Overlaying HYSPLIT back trajectories

“Interoperability”: The ability of diverse systems and organizations to work together to exchange and utilize information.

To facilitate this, VIEWS will offer the following services and features:

• Discovery, retrieval, and exchange of data and metadata

• On-the-fly transformation and formatting of data and metadata

• Upload and management of user-supplied data and metadata

• Generation of visualization and analysis products

• Availability of “embeddable” components and tools

• Support of OGC web standards for spatial data exchange

VIEWS/TSS Interoperability: Services and standards

Earth Science Information Partners (ESIP) Federation•Collaborate on interoperability efforts for data/metadata discovery and exchange

•Contribute data and services to the Earth Information Exchange

•ESIP website: http://www.esipfed.org/

GEO/GEOSS Architecture Implementation Pilot (AIP)•Contribute components and services to the GEOSS registry

•Participate in efforts to standardize and streamline data discovery and exchange

•Develop VIEWS/TSS as an “persistent operational exemplar” of the GEOSS architecture

•GEOSS AIP 2 link: http://www.ogcnetwork.net/AIpilot

CyAir (cyberinfrastructure for the air quality community)•Contribute to the planning, development, maintenance and coordination of systems to help the air quality community better utilize air quality data and information.

•Assess the current state of air quality data providers

•Investigate and assess data user needs and issues, including types of users, types of data, data access methods, data usage and transformation, and software usage

•Identify and assess existing or needed standards and conventions

•Generate a community vision for air quality cyberinfrastructure

Interoperability and Collaborations

VIEWS/TSS Data Value Chain Diagram

Roadmap: From now until the project ends

January 2011 – Beta testing phase begins

• January 10-11: Beta tester’s meeting at UNC• January-May, 2011:

– Organize and prioritize feedback– Incorporate ideas and requests– Resolve problems and issues– Import additional data– Generate additional derived products– Implement key case studies with component tools– Push changes and additions to testers and users– Repeat the above steps in various combinations

• Solicit NASA for a possible project extension

May 2011 – Period of performance for the project ends

• Publish final versions of data products, website tools, and project reports• Organize and conduct any final training and/or project status meetings• Announce enhancements via VIEWS and TSS websites and user newsletter• Design scope of work and strategy for project extension, if forthcoming

Roadmap: Keys issues, operation and maintenance

VIEWS/TSS funding has expired, and no renewals have been tendered

• All work on the project, including ongoing data import, will cease after May 2011• Responses to feedback and support requests will cease or be intermittent and/or

infrequent• No hardware repairs will be made• No software upgrades or troubleshooting can be done• The websites will “piggyback” on existing hardware and infrastructure (electricity)

as long as possible

NPS is funding the Federal Environmental Database (FED)

• Leverages key components and infrastructure from VIEWS• Will continue the import of ground-based datasets• Will improve and refine various tools• But no funding for satellite-related data or development

Possible extension to NASA ROSES project

• Will help us finish key functionality and polish existing features• Will extend the time over which new satellite and modeled data will be imported• But it’s not a sure thing

The End…

Thanks!