Upload
maria-stone
View
219
Download
2
Tags:
Embed Size (px)
Citation preview
ESA Data Integration ApplicationOpen Grid Services for Earth Observation
Luigi Fusco, Pedro Gonçalves
Topics
• Earth observation requirements
• Ozone application
• Generic application layer (GridEngine)
• Grid portal for EO services
Requirements
• 100 Gbytes of data per day in ERS missions and 500 Gbytes in ENVISAT
• Grid Requirements :– enhance the ability to access high
level products– allow reprocessing of large historical archives– improve Earth science complex applications (data
fusion, data mining, modeling …)
Ozone Application
• Input: Wave spectra data measured by the GOME instrument on the ERS (level 1)
• Output: Calculation of satellite ozone profiles (level 2 data)
• Two algorithms: OPERA (MPI) and NOPREGO (Neural Networks)
• Objective: Data validation using LIDAR measurements
• Collaboration among different institutes: France (IPSL), Italy (ESA, ENEA, UTV), Holland (KNMI)
Ozone Application Requirements
• Reuse of Legacy data warehouses and catalogues from ESA
• Constraints:1. Access to DataGRID storage elements and execution
of remote jobs within DataGRID• GLOBUS 2.2.4
2. CEOS-GRID • GLOBUS 2.4.2 • Different certificate authorities
• Web portal interface
Ozone Application Portal• Temporal and spatial selection of data • Catalogue access and data transfer from ESA
data warehouses to the GRID storage elements• Job selection and
status information• Result retrieval and
visualization in OWS• Remote MySQL
access (SOAP)• Data validation w/
ground measurements• Demonstration of
prototype at last WGISS meeting in Toulouse
Lessons learned … Joining the services
Client Applications (JAVA / Linux / Windows)
Web Portal
Client ApplicationsSO
AP
W
rap
per
SO
AP
W
rap
per
OG
C W
eb
Serv
ices
MUIS ESA Catalogue
AMS ESA Data Archive
WCS / WFS / WMS
Catalogue (CSS)
GRID EngineDATAGRID
Computing
Elements
Storage Elements
GLOBUS 2.2
Computing
Elements
Storage Elements
… other versions
Computing
Elements
Storage Elements
GRID Engine
Status : First version for DataGrid and GLOBUS 2.2
• Connects and manages jobs sessions in multiple GRID environments - (WSDL description and SOAP encoding)
• Discovery Services• Publish jobs availability and current processing
status to external services in XML• Retrieves and translates job results GRID Engine
DATAGRID
Computing
Elements
Storage Elements
GLOBUS 2.2
Computing
Elements
Storage Elements
GLOBUS 3
Computing
Elements
Storage Elements
GRID Engine
• Services are pre-defined applications managed
by templates for each GRID that are plugged into the Grid Engine
• Templates instantiate the necessary scripts for the specific job requests (files names, parameters)
• Defined by XML schema for future implementation of discovery services
GRID Engine - Operations
• SOAP Services using HTTP(S)
• Session Manager (SM)– Session and Certificates management – SM_open, SM_authenticate, SM_quit, …
• Job Manager (JM)– Job commands and status information – JM_submit, JM_list, JM_create, …
• Storage Explorer (SE)– Storage access – SE_query, SE_move and SE_transfer methods. TBD : the use of SOAP w/ Attachments (W3C sub.)
Example of EO specialized Services
• On-demand geographical orbit projection (using ESA’s EnviProj)
• On-demand orbit projection and mosaicking (using OSSIM)
• Format Conversion Services: Envisat standard, GeoTiff, HDF-EOS … (using geoTiff, OSSIM and with the future use EnviView libraries)
• GOMOS/GOME level 1 processing and validation (using ESA data processors)
EO data Search and Storage Services
• Temporal and spatial selection of data
• Data transfer from data warehouses (AMS)
• Data transfer to and from the GRID storage elements
Auxiliary Grid Services
• Job selection, launching and status information• Result retrieval and
visualization in WMS• Join personal data
folder as a WMS
Conclusion
• The Web Portal Integrates several components of the EO Infrastructure
• The EO GRID Engine provides a standard way for Web Clients to request GRID operations
• The Web Portal and the EO Grid Engine allow the complexity of the underlying components to be hidden from the user
• This Framework can be adapted to provide new tools for the satellite end-users (e.g. integration of ENVISAT specific tools, such as BEAT, BEST, BEAM …)