Workflow languages and Workflow languages and engines breakoutengines breakout

Matthew AddisMatthew Addis

IT InnovationIT Innovation

5 December 20035 December 2003

NeSC workshop on workflow servicesNeSC workshop on workflow services

Workflow languages and Workflow languages and engines breakoutengines breakout

• Objective:Objective:– Better understand the requirements for workflow Better understand the requirements for workflow

languages and engines in scientific applicationslanguages and engines in scientific applications– Allow comparison of work already done by Allow comparison of work already done by

projects employing scientific workflow/dataflowprojects employing scientific workflow/dataflow

• Approach Approach – Identify the different areas/types of requirementsIdentify the different areas/types of requirements– Identify the different levels at which these Identify the different levels at which these

requirements might exist using a ‘stack’ type requirements might exist using a ‘stack’ type approachapproach

Requirements areasRequirements areas

• PerformancePerformance• SchedulingScheduling• DiscoveryDiscovery• Events/monitoring/reportingEvents/monitoring/reporting• Fault toleranceFault tolerance• ScalabilityScalability• Launching/invocation/executionLaunching/invocation/execution• Steering/interaction/controlSteering/interaction/control• ManageabilityManageability

StackStackGoals

Abstract flows

Concrete flows

Process execution

Service descriptions

Messages

Communications

Policies and security

Ontologies, m

etadata, descriptions,inform

ation services

Positioning of existing Positioning of existing systemssystems

Goals

Abstract flows

Concrete flows

Process execution

Service descriptions

Messages

Communications

Freefluo

VDL,Taverna,

scuflDagman/Condor

Triana

Iceni

Some characteristics to Some characteristics to consider when differentiating consider when differentiating

existing approaches existing approaches

• Execution policies/approaches, e.g. data flowsExecution policies/approaches, e.g. data flows• Models and structures: e.g. DAGsModels and structures: e.g. DAGs• Data model and typesData model and types• We need to pull together and reuse existing We need to pull together and reuse existing

body of work in this areabody of work in this area

Plumbing groupPlumbing group• Streams are important in scienceStreams are important in science

– Not supported in commercial systemsNot supported in commercial systems

• DependabilityDependability– Detection of failuresDetection of failures– Propagation of exceptionsPropagation of exceptions– HandlingHandling

• OptimisationOptimisation– Quality of serviceQuality of service– Networks, computers, sets of resourcesNetworks, computers, sets of resources– Choices of data formats and transfer mechanismsChoices of data formats and transfer mechanisms

• ManageabilityManageability– Monitor what’s going onMonitor what’s going on– Control over execution and servicesControl over execution and services– Dynamic adaptation of workflowDynamic adaptation of workflow

Next stepsNext steps

• Establish email discussion groupEstablish email discussion group• More work on describing existing systems More work on describing existing systems

with respect to the areas we’ve identifiedwith respect to the areas we’ve identified• ReportReport• Ideally:Ideally:

– Common research and developmentCommon research and development– What is the smallest reference architecture that What is the smallest reference architecture that

satisfies most of what we wantsatisfies most of what we want