EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE EGEE and gLite are registered trademarks INFSO-RI-508833 Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE and gLite are registered trademarks INFSO-RI-508833


The Wisdom Environment

Vincent Bloch CNRS-IN2P3

ACGRID School

Hanoi (Vietnam) November 8th, 2007

Credits: Jean Salzemann

2



WISDOM initiative

• WISDOM initiative aims to demonstrate the relevance and the impact of the grid approach to address drug discovery for neglected and emerging diseases.

• First achieved experiences:– Summer 2005: Wide In Silico Docking On Malaria (WISDOM)– Spring 2006: Accelerate drug design against H5N1 neuraminidase– Winter 2006: Second data challenge on Malaria

• Partners:– Grid infrastructures: EGEE, Auvergrid, TWGrid, EELA, EuChinaGrid,

EuMedGrid– European projects: Embrace, BioinfoGrid, EGEE– Institutes and association: Fraunhofer SCAI, Academia Sinica of

Taiwan, ITB, Unimo University, LPC, CMBA, CERN-ARDA, HealthGrid

3



Challenges for high throughput virtual dockingExample: data challenge against H5N1 NA

300,000 Chemical compounds:ZINC &Chemical combinatorial library

Target (PDB) :Neuraminidase (8 structures)

Millions of chemicalcompounds availablein laboratories

In vitro high Throughput Screening1$/compound, nearly impossible

Molecular docking (Autodock)~100 CPU years, 600 GB data

Data challenge on EGEE, Auvergrid, TWGrid~6 weeks on ~2000 computers

In vitroscreeningof 100 hits

Hits sorting and refining

N1H5 N1H5

4



Issues for the grid-enabled high throughput virtual docking

• Computer-based in-silico screening can help to identify the most promising leads for biological tests– Involve whole databases (ZINC)– reduces the cost of trail-and-error approach

• In silico docking is well-fitted for grid deployment– CPU intensive application– Huge amount of output– Embarrassingly Parallel

• Issues of a large scale grid deployment– The rate of submitted jobs must be carefully monitored– The amount of transferred data impacts on grid performance– Grid process introduces significant delays– Licensed software requires licenses distribution strategy on grid

5



Grid tools used during the data challenges

• WISDOM– a workflow of grid job handling: automated job submission,

status check and report, error recovery– push model job scheduling– batch mode job handling– http://wisdom.eu-egee.fr

• DIANE– a framework for applications with master-worker model– pull mode job scheduling– interactive mode job handling with flexible failure recovery

feature– http://cern.ch/diane

6



Grid components interacting with WISDOM

• The WMS:

– The user submits jobs via the Workload Management System – The Goal of WMS is the distributed scheduling and resource

management in a Grid environment.– What does it allow Grid users to do?

To submit their jobs To get information about their status To cancel them To retrieve their output

– The WMS tries to optimize the usage of resources as well as execute user jobs as fast as possible

7



WMS Components

WMS is currently composed of the following parts:• User Interface (UI) : access point for the user to the WMS • Resource Broker (RB) : the broker of GRID resources,

responsible to find the “best” resources where to submit jobs

• Job Submission Service (JSS) : provides a reliable submission system

• Information Index (BDII) : a server (based on LDAP) which collects information about Grid resources – used by the Resource Broker to rank and select resources

• Logging and Bookkeeping services (LB) : store Job Info available for users to query

8



Grid components interacting with WISDOM

• DMS: Data Management system– The user can store files on the grid through the DMS.

– The goal of the DMS is to virtualize data on the grid and guarantee security integrity, and reliability of the data

– What it allows Grid users to do: Copy Files on the Grid Register files on the Grid with a logical name Store and manage metadata related to a file Replicate files on the Grid Delete files on the Grid Retrieve files from the Grid

9



DMS Components

• LFC (LCG File Catalogue):– It is used to register files on the grid– LFC provides functionalities to give logical names to files and

organize them in directories

• GridFTP:– Low level file transfer protocol – Secured and reliable

• AMGA:– It is an grid interface for relational databases– Can be used to store medata– Can be used as a file catalogue

10



other components interacting with WISDOM

• VOMS (Virtual Organisation Membership Service)– Store information concerning VO and roles

• FlexLm floating licenses server

• Web Portals– Can be used to visualize statistics or results

• Remote Database Servers– Can be used to store some information remotely (results,

metadata etc..)

11



WISDOM technology

• WISDOM has been specifically developed around EGEE middlewares (LCG-2.7 , Glite).

• It uses a Java Multithreaded submission Engine

• Main scripts are written in perl

• Job-related scripts in written shell script (bash)

• Future environment will include– Web Services technology (WS-I profile)– Java and Python AMGA clients– All the code written in Java– Security and fine-grained ACLs

12



• 2 main scripts:– wisdom_submit:

submits the jobs with a java multithreaded submission engine stores the job ID and command lines and store them in a database.

– wisdom_status:

checks the status of jobs regularly handle the resubmissions of failed, aborted and cancelled jobs. reads the IDs from wisdom_submit database stores the job IDs in a table to prevent crushing wisdom_submit

files, along with other parameters:• job number• a submission job status (unsubmitted, submitted, done)• job submission count.

The process will loop until all the jobs of the instance are not finished.

WISDOM ENVIRONMENT (1/2)

13



WISDOM ENVIRONMENT (2/2)

• Several Features:– No input and output sandboxes in jobs.

All the target files, ligands and software are copied dynamically from the SE to the WN to unload the RB i/o.

FlexX outputs and Grid outputs are saved on several SEs through LFC and GridFTP

– Jobs JDL and scripts are generated just before any submission to take the wisdom.conf modifications into account (CE and RB black

lists, job submission frequency) are deleted afterward to save disk space.

– Dynamic insertions of docking results and statistics in databases which allow real-time visualisation of the DC status.

– wisdom_status can be stopped at any given time and restarted: it saves its own memory environment, so it can be restarted after a crash.

14



Instance Definition

• The instance is a set of jobs regrouped accordingly to different criteria.

• The instance is unique, and has its own name• The instance is submitted entirely on the grid, then it is

followed up• The instance name is by default: • <TARGET><PARAMETER><DATABASE>

• Instance’s jobs are called after the instance name:

• <INSTANCE NAME>J<number of the job>

15



WISDOM deployment

GRIDGrid services (RB, RLS…)

Grid resources (CE, SE)Application components

(Software, database)

installation

Installer Tester

Test the grid

wisdom_executionWorkload definition

Job submissionJob monitoring

Job bookkeepingFault trackingFault fixing

Job resubmission

Set of jobs

User

Collection

Accounting data

Superviser

Web sitedatabase

License server

16



WISDOM Integration example

DMS/GFTP

User Interface

HealthGrid Server

Web Site

WMSSEsCEs &WNs

User Interface

Wisdom_submit

Wisdom_status

WMSSubmits the jobs

Checks job status resubmits

CEs &WNs

FlexXjob

SEs

Structure file

Compounds file

inputs

outputs

Output file

Web Site WISDOMDB

OutputDB

Docking information

Statistics

FlexX

Statistics Flexlm

server

Flexlm

server

Flexlm

server

17



Environment architecture (1/2)

• wisdom.conf – the file that define the configuration of the instance

• wisdom_submit.sh – the execution script that launch the instance submission

• wisdom_submit.pl – the perl script of the execution process which submits the

instance

• wisdom_status.sh – the execution script that launches the instance status checking

• wisdom_status.pl – the perl script of the status checking

18



Environment architecture (2/2)

• bin/flexx.sh (the flexx script that is run by the jobs)• bin/mt-job-submit (the execution script of the multi-threaded

submission engine) • bin/MTJobSubmitter.jar (the jar file of the multi-threaded java

submission engine)• bin/checkit.sh (a script used at the end of jobs to check the status

of the job and store everything on the grid)• bin/lfc_env.sh (a script to set up the environment variables for

LFC)• input/<DATABASE>/db_urls* ( there are several files, 1000, 2000,

3000, 4000… each of these files has the sfn of the database subsets replica. It is used in case of failure of the LFC server).

• edg_wl_ui_config/* (this directory hold all the configuration files of the resource brokers)

• Files need to be edited accordingly to the application and the VO!

19



Simplified grid workflow for WISDOM

StorageStorageElementElement

ComputiComputingngElementElement

Site1

Site2

StorageStorageElementElement

User interfaceUser interface

ComputiComputingngElementElement

Compounds database

Parameter settingsTarget structures

Results

Results

Statistics

ResourceResourceBrokerBroker

Software

WISDOM production system

Jobs

Subsets

Subsets

20



WISDOM and Security

• Instances are submitted by a given user– All the jobs of the instance are belonging to the same user– Resources are dependant on the user’s VO

• Outputs files – Stored on the VO storage elements and register with the VO

LFC.– If LFC is failing, files are stored on the VO writable directory on a

given storage element

This implies that: • Users must follow-up the execution, and need to renew

their proxy if necessary• Files stored, are, a priori, available to all the VO

members

21



The new environment

• Web Services Interface– better interoperability– Everything is controlled through a few set of operations (no more

modification of the files are required)

• Dynamic storing/querying of the results and jobs information on the Grid using AMGA metadata management system

• Improved fault tolerance

• Improved flexibility – New applications can be deployed more easily– As well as corresponding data

• Secured and multi-user

22



The new environment

• Entirely developed using Java– No need to use text files to send information between the

submission and the monitoring of the jobs– Improved fault tolerance– Improved flexibility– New application easier to deploy

• Improved monitoring of the grid resources– Uses its own ranking based on BDII information– Takes into account the number of jobs submitted to the sites to

avoid overloads (the jobs are sent where the free CPUs are)– Takes into account the jobs failures and failing reasons (the

“bad” sites are penalized)

23



The new environment

• Uses AMGA for jobs and data monitoring– Improved monitoring and statistics– Dynamic storage and query of the data and results– Allows “Pull Model”

• Web Services Interface– Better interoperability– Ease the access to the environment: everything is controlled

through a few set of operations

24



New environment process

• Retrieve BDII information concerning the CE (number of CPUs, free CPUs,…)

• Define a workload according to the CE information• Initialize the voms proxy• Generate the jobs JDL• Submit the jobs using multithreaded submission• Until all the jobs are successful:

– Check the status of the jobs using multithreaded check– Resubmit jobs if needed– Re-initialize voms proxy if needed– Update instance information in AMGA

25



New Environment Architecture

26



hits

hits_id

rank

simulation_id

energy_level

run

mean_energy

Cluster_count

simulation

dlg_file

simulation_id

Target_id

Ligand_id

Histogram_file

file

coordinates_blob

hits_id

ligands

Ligand_id

Library_id

name

pdbq_file

mass

psa

logp

donor

acceptor

Logd7_4

ring

rb

far

atoms

refra

library

Library_name

library_id

target

name

Target_id

pdbqs

maps

1,1

1,n

1,1

1,1

1,1

1,n

1,1 1,n1,1

1,n

project

description

Project_id

Program_name

Program_version

Program_options

Project_id

1,n

1,1

coordinates_file

Agent_id

job

Result Database Schema (autodock)

27



Monitoring schema

28



Pull Model

• Instead of sending a task with the job, the job retrieve a task from the task database while running

• The job performs tasks as long as it is running• Pros:

– No need to define a workload before the job submission– No need to have all the jobs running– When a job fails, only the last task need to be recomputed

• Cons:– Need to store the results on the fly– No access to the output sandbox– Retrieving a task can increase the job overhead

29



Questions?

Documents

EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE EGEE and gLite are registered trademarks INFSO-RI-508833 Enabling Grids for E-sciencE