David Adams

ATLAS

AJDL: Analysis Job Description Language

David AdamsBNL

December 15, 2003

PPDG Collaboration MeetingLBL

AJDL PPDG All Hands – uJDL session December 15, 2003 2

David Adams

ATLAS

Contents

Analysis

AJDL

Data object types

Service functions

Conclusions

AJDL PPDG All Hands – uJDL session December 15, 2003 3

David Adams

ATLAS

Analysis serviceFunctionality

• User specifies application, task and dataset

• Service processes and returns result

• User examines result, modifies task or changes dataset and resubmits

• User monitors job during processing

• User has option to kill job

Common interface• Provides the above

• Allows generic service to be shared– Robust, efficient analysis service is difficult

AJDL PPDG All Hands – uJDL session December 15, 2003 4

David Adams

ATLAS

UserAnalysis

Job 1

Job 2

Application Task

Dataset 1

Analysis Svc

1. Create or locate

2. select 3. Create or select

4. select

5. submit(app,tsk,ds,cfg)

6. splitDataset

Dataset 2

7. create

e.g. ROOT

e.g. athena

Result9. fill

10. gather

Result 9. fill

Result CodeAnalysis service

AJDL PPDG All Hands – uJDL session December 15, 2003 5

David Adams

ATLAS

AJDL as a bridge

P I/S E A L

G A N G A

R O O T

J A S

C o m m a nd line

A J D L

P R O O FA R D A

C o nd o r-G

G C E /C him e raS T A RJ D A P (J A S )

D IA L-inte ra c tiv e

An a ly s is e n v iro n m e n ts S ch e d u le r s

G r idse rv ice s

A T LA S p ro d u c tio n

P lu g -inclie n ts

P o rta l/s w itc h

AJDL PPDG All Hands – uJDL session December 15, 2003 6

David Adams

ATLAS

AJDLAnalysis Job Description Language (AJDL) includes

• Data object types– E.g. could be expressed in XML

• Analysis service functions taking objects as arguments– E.g. SOAP messages

For now, we are more concerned with the content of these objects and messages

• Rather than the format

• However, need format for implementation

DIAL WSDL and DTD is close

AJDL PPDG All Hands – uJDL session December 15, 2003 7

David Adams

ATLAS

Data object typesApplication

Task

Dataset (and subtypes)

Configuration

Result (and subtypes?)

Job

AJDL PPDG All Hands – uJDL session December 15, 2003 8

David Adams

ATLAS

ApplicationPurpose

• Specify the application executable

Content• Name and version (of application SW package)

Analysis service usage• Locate script to build task

• Locate script to process dataset and generate result

Package management service• Provides location of app package (name, version)

• May install app package and dependencies– Exe is likely in separate package

AJDL PPDG All Hands – uJDL session December 15, 2003 9

David Adams

ATLAS

TaskPurpose

• Allow user to configure application

Content• Collection of named files

– Embedded text or

– Logical files

• Files contain parameters and code

Analysis service usage• Use with script to build task

• Stage files for building task

AJDL PPDG All Hands – uJDL session December 15, 2003 10

David Adams

ATLAS

DatasetPurpose

• Specify and split data for processing

Content• Description used by application processing script

– Dataset subtype

• Event count and ID list– EventDataset

• List of constituent datasets– CompoundDataset

• Abstract dataset specification– VirtualDataset (no location)

– QueryDataset

AJDL PPDG All Hands – uJDL session December 15, 2003 11

David Adams

ATLAS

Dataset (cont)Analysis service usage

• Used with script to process data and generate result– Pass dataset to script

• Split dataset to create sub-jobs– EventDataset to split along event boundaries

– CompoundDataset provides guidance for splitting

• Find concrete dataset for abstract specification– Use dataset selection service for QueryDataset

– Use dataset replica service for VirtualDataset

AJDL PPDG All Hands – uJDL session December 15, 2003 12

David Adams

ATLAS

ConfigurationPurpose

• Specify how to process dataset

Content• User identity and role

• Response time– Desired

– Maximum acceptable

• Accounting– Account to charge

– Budget

AJDL PPDG All Hands – uJDL session December 15, 2003 13

David Adams

ATLAS

ResultPurpose

• Communicate results to user

Content• Datasets

• Named files– Embedded or logical

– For arbitrary data

> E.g. collection of histograms

– Or use Result subtype?

AJDL PPDG All Hands – uJDL session December 15, 2003 14

David Adams

ATLAS

JobPurpose

• Communicate job status and results to user

Content• Application, task, dataset and configuration

• Result– Possibly partial

• Status– Running, done, failed, …

• Start, stop and update times

AJDL PPDG All Hands – uJDL session December 15, 2003 15

David Adams

ATLAS

Service functionsAnalysis service includes

• bool has_application(Application)

• bool has_task(Application, Task)

• bool install_task(Application, Task)

• JobId submit(Application, Task, Dataset, Config)

• Job job(JobId)

• bool kill(JobId)

AJDL PPDG All Hands – uJDL session December 15, 2003 16

David Adams

ATLAS

ConclusionsStandard analysis service interface

• Allow common service to be shared

• First pass presented here

More information• AJDL document

• DIAL JDL page