MLalign2D on the Grid

MLalign2D on the Grid

Germán Carrera, Alfredo Solano (CNB/CSIC)

EMBRACE COURSE

Monday 19th of February to Friday 23th. CNB-CSIC Madrid

MLalign2D on the Grid

• What we have to do to run MLalign2D on the EGGE Grid?

– We have ported MLalign2D and an early version of MLrefine3D to the Grid:

– This software uses the DIANE framework to launch the jobs to the Gird and a program called GANGA for the jobs management.

MLalign2D on the Grid

• This presentation pretends to be an explanation of how to run this application on the EGEE Grid.

• 1 Previous steps – 1.1 Accessing to the UI (User Interface)

• (eg:)ssh -X user@machine • user is the username • machine is the name of the UI

MLalign2D on the Grid

• 1.2 Openning a Grid proxy – (eg:)voms-proxy-init --voms biomed --valid 50:0

– biomed is the virtual organization (vo)

– --valid 50:0 is an optional flag that shows the expiration time of the proxy (50 hours in this case)

– (eg:)voms-proxy-destroy (To close the current proxy)

– (eg:)voms-proxy-info --all (To consult diverse information about your opened proxy)

MLalign2D on the Grid

• 1.3 Uploading your images to the UI (User Interface) – (eg:) scp -r directory user@machine:directory

– directory is the name of the directory that contains your images

– user is the username

– machine is the name of the UI

– directory(2) is the name of the directory that will be created on the user interface

MLalign2D on the Grid

• 2 Configuring DIANE for xmipp_MLalign2D and xmipp_MLrefine3D – 2.1 Adding the paths of GANGA/DIANE and

xmipp_MLalign2D/xmipp_MLrefine3D binary files • The path of GANGA binaries in villon machine is

/opt/ganga/install/slc3_gcc323/4.2.1/bin • The path of DIANE binaries in villon machine is

/opt/diane/specific/slc3_gcc323/HEAD/DIANE/dev/commands • The path of the MLalign2D and MLrefine3D binary files is /opt/xmipp • (eg:) export PATH=/opt/ganga/install/slc3_gcc323/4.2.1/bin:$PATH • (eg:) export

PATH=/opt/diane/specific/slc3_gcc323/HEAD/DIANE/dev/commands:$PATH

• (eg:) export PATH=/opt/xmipp:$PATH • You can add the previous lines to the .bashrc file (is placed in your home

directory) and run (eg:) source .bashrc - to fix the changes

MLalign2D on the Grid

• 2.2 DIANE/xmipp_MLalign2D xmipp_MLrefine3D ".job" configuration files

• 2.2.1 Coping a ".job" file to our home directory – (eg:) cp

/opt/diane/specific/slc3_gcc323/HEAD/DIANE/dev/workspace/xmipp_test.job $HOME

MLalign2D on the Grid

• 2.2.2 Description of the ".job" file tags – Directory With this variable we indicate the directory in which we have the ".sel"

file and the images that it references – (eg:) dir = '/home/user/imagesdir/' Application With this variable we indicate that

we want to use the xmipp_MLalign2D application – (eg:) Application = 'xmipp' Data (MLalign2D) Data var contains different tags, the

description of each one is: – Program we can choose between xmipp_MLalign2D and xmipp_MLrefine3D – Niter contains the number of iteractions – imgselfile contains the path to the ".sel" file – inirefsel contains the path to the reference images – add_params to add more params to the execution – MLroot Rootname for all output files – noise – offset – Nsplit split the full set of images in different subsets – iter Number of iteractions of each subset of images (allways at 1)

MLalign2D on the Grid

Data = {'Niter' : 2,

'Program' : 'MLalign2D'

'imgselfile' : dir+'selfile.sel',

'inirefsel' : dir+'refselfile.sel', # CREATE THE REFERENCE IMAGES FIRST!

'add_params' : '',

'MLroot' : 'output',

'noise' : 0,

'offset' : 0,

'Nsplit' : 3, # SPLIT ALL IMAGES IN A NUMBER OF GROUPS

'iter' : 1 }

MLalign2D on the Grid

JobChain It references the Data var (eg:) JobChain = [Data] JobInitData Init data for the execution of

DIANE/xmipp_MLalign2D jobs (eg:) JobInitData = None (eg: MLalign2D case) InputFiles = [Data['imgselfile'],Data['inirefsel'],'/opt/xmipp/xmipp_MLalign2D'] (eg: MLalign3D case) InputFiles =

[Data['imgselfile'],Data['inirefsel'],'/opt/xmipp/xmipp_MLrefine3D'] InputFiles Input files needed for the execution of DIANE/xmipp_MLalign2D jobs

(eg:) InputFiles = [Data['imgselfile'],Data['inirefsel'],'/opt/xmipp/xmipp_MLrefine3D'] * '/opt/xmipp/ is the path of some xmipp programs in our user interface

MLalign2D on the Grid

• 2.3 Running DIANE with xmipp_MLalign2D and xmipp_MLrefine3D – tcsh – source /opt/diane/specific/slc3_gcc323/HEAD/DIANE/etc/environment.csh – diane.startjob -j $PWD/xmipp_test.job -w5@LCG --ganga Where xmipp_test.job

is the file we have defined before – -w5@ (eg: 5) is the number of workers (CPU's on the Grid), LCG is the backend – --ganga is the management and job submission tool – If all goes well you have to obtain something like that: DIANE: 11:43:40: job

output in: /home/gcarrera/diane.workspace/jobs/ – You can see the status of your jobs usign the “ganga” utilit– You can add new workers with the command (eg:) diane.ganga.submitworkers –

job=xmipp_test_gem.job –nw=10 –bk=lcg– When DIANE execution has finished you can consult diverse information

accessing to the MonALISA Repository at http://lcgui003.cern.ch:8080/