Upload
myron-hopkins
View
223
Download
0
Embed Size (px)
Citation preview
INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
Project Gridification: the UNOSAT experience
Patricia Méndez Lorenzo
CERN (IT-PSS/ED)
CERN, 1st March 2006
EGEE USER FORUM
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Outlook
◘ UNOSAT Presentation
◘ Collaboration with CERN
◘ UNOSAT and GRID
◘ Production of UNOSAT inside the Grid environment
◘ Future Plans
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
UNOSAT Presentation
UNOSAT is a United Nations Initiative
◘ Objectives➸ Provide the humanitarian community with access to
satellite imagery and Geographic Information System services
► Reduce disasters and plan sustainable development
➸ Ensure cost-effective and timely products
◘ Core Services➸ Humanitarian Mapping➸ Image Processing
VEGETATION – 1 Km
IKONOS – 1m
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
UNOSAT Service Chain
Data suppliers
Ground station
USER
UNOSATCentral Unit
WWW
a) Detailed satellite maps of the area are required by UNOSAT Satellite responds the requirement with radar and optic techniques
b) Satellites send raw data back to Earth
c) Advanced data processing performed
d) Data relayed by UNOSAT
e) Additional processing anddownload in the web pageof UNOSAT
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Relief Projects of UNOSAT
◘ Case Study: Indian Ocean Tsunami Relief and Development◘ 29th Dec 2004: First Map distributed online to field users
➸ 14th Jan 2005: Imagery Bank online:► 100 Tsunami-related maps (pre and post)► 670 raw satellite images
➸ January: 200,000 tsunami maps downloaded in total
◘ After the Tsunami catastrophe
➸ Pakistan
➸ Filipinas
◘ In both cases maps are hosted at CERN
◘ Projects founded by European
partners as GMES (Global Monitoring Environment and Security)
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
First step: UNOSAT and CERN
◘ UNOSAT is CERN partner since 2002
◘ CERN supports them with network facilities, with computer
infrastructure and with human (support) resources
◘ Asian Tsunami Example: ➸ Central Web Services at CERN under considerable strain
➸ Availability of the UNOSAT data at any time
◘ UNOSAT provides the users with a web interface able to find
the files of the images by clicking on the earth images
➸ Attractive method, easy to use
◘ Something similar to do with the GRID ➸ One of our goals inside the UNOSAT-GRID Collaboration
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
One step further: GRID
◘ Potential Bottlenecks: ➸ UNOSAT beginning to suffer from limited capacity and processing power ➸ Multiple satellites being launched ➸ Larger and larger storage capacity needed
◘ In summer 2005 we have provided a whole structure at CERN for
UNOSAT➸ UNOSAT Virtual Organization (VO)➸ 3.5TB in CASTOR➸ Computing Elements, Resource Brokers ➸ Collaboration with ARDA group➸ AFS area of 5GB
◘ We have run some UNOSAT tests (images compression) inside the
GRID environment (quite successful)
◘ The framework developed for in principle for Geant4 (See Alberto
Ribon’s presentation [49]) has been adapted for UNOSAT needs
We have provided the whole GRID infrastructureAt CERN
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
UNOSAT Production inside GRID
◘ UNOSAT provided us with a set of images for testing
◘ Associated to each image a metadata file was included
File name, directory path, geographical coordinates
◘ Steps:
STORAGE LEVEL
➸ Copy and registration of the images in Castor@CERN
► Use of the LFC Catalog ➸ Parse the metadata files to extract the different metadata ➸ Use of the AMGA tool to parse metadata to location of the files
COMPUTING LEVEL
➸ Use of compression tools to compress images inside LCG resources ➸ Use of the general submission tool adapted to UNOSAT needs
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
A GRID Metadata Catalogue
◘ LFC Catalogue➸ Mapping of LFN to PFN
◘ UNOSAT requires➸ User will give as input data certain coordinates
➸ As output, he wants the PFN for downloading
◘ The ARDA Group assists us setting up the AMGA tool for UNOSAT
Oracle DB
ARDA APP
LFC
CASTOR
SRMMetadata
(x,y,z)LFN PFN
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Tools developed for new Gridifications
Generation of a general framework consisting of 2 major tools:
1. Tool to perform the automatic job submission 2. Tool to retrieve and handle the corresponding output
1. Automatic job submission◘ Given an user’s jdl this tool performs the following actions: ➸ It lists all sites able to run the jdl provided by the user ➸ It creates automatically a jdl file based on that provided
by the user ➸ It submits the just created jdl containing the user
application(s) ➸ Moreover it creates a subdirectory (defined by the user)
containing a list of the sites where the jobs have been submitted, the corresponding jdls and the jobs IDs
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Tools developed for new Gridifications
◘ Additional Features: ➸ The user can define the queues where the jobs are submitted. These queues are checked to see whether it fixes the job requirements. ➸ Requested LFN files can be included. The corresponding
TURLs are searched and included in a file passed in the InputSandbox to the WN
◘ Applications ➸ This tool has been used for the 1st and the 2nd phases of the production: software installation and production
◘ Usage: ./submitter_general -vo geant4 -jdl jdlexample -jobfile G4_PROD -data /grid/geant4/production_software
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Tools developed for new Gridifications
2. Retrieve and handle of the outputs
➸ The 2nd tool checks the status of the jobs from the job IDs included in the directory given by the user
◘ Usage: ./get_output -jobfile G4_PROD -dest G4_PROD/outputs
◘ OutputThe job run in ramses.dcic.ups.es:2119/jobmanager-torque-
dteam is in status: ScheduledThe job run in grid01.phy.ncu.edu.tw:2119/jobmanager-torque-
dteam is in status: runningThe job run in scaic10.scai.frauhofer.de:2119/jobmanager-
torque-dteam is in status: over
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Tools developed for new Gridifications
◘ Additional Features:
➸ It is possible to
visualize the
outputs on the web
➸ A html report is
provided
showing the files
decided by the user
CERN, 1st March 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Future Plans with UNOSAT
◘ Collaboration between UNOSAT, ARDA and GD➸ 1(2) ARDA and 2 UNOSAT Students
➸ Still many discussions needed
➸ Support from other sites foreseen
(x,y,z)GRID WORLD
Application
◘ User can get the info in his laptop too
◘ Fundamental AMGA