Sphinx Server

Sphinx Client

Data Warehouse

Submitter

Generic Grid Site

Monitoring Service

Resource

Message Interface

Current Sphinx Client/Server

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Message Interface

Prediction Engine

Sphinx Server

Sphinx Client

Data Warehouse

Submitter

Generic Grid Site

Monitoring Service

Resource

Message Interface

Sphinx Client/Server with V2 Components

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Message Interface

Sphinx Server

Data Warehouse

Stand alone Sphinx Server with light Client

Sphinx Client

Generic Grid Site

Monitoring Service

Resource

Message Interface

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

Sphinx Server

Data Warehouse• Message Tables• Policy Information Tables• Entity Accounting Tables• Work (DAG, Job, etc) Tables

• State• Unreduced• Unpredicted• Unaccepted (not admitted)• Unplanned (not yet feasible)• Unsent (assigned)• Unfinished (executing)• Remove (accounted or rejected)

• Entity• Resource Property Requirements• Dependency Requirements

• Do not start before “event”• I/O

• QoS Requests• Resource Assignment for next k steps

• Application Tables (in general, distributed…)

• Transformation Catalogue• Profiles (CPU, Disk, Bandwidth

requirements)• History (CPU, Disk, Bandwidth, actual

use)• Data Tables (in general,

distributed…)• Replica Catalogue (from RLS)• Profiles (Size, Bandwidth

requirements)• History (Access rate, etc)

• Resource Property Tables• CPU, Disk, Network Connection,

middleware configuration, etc• Grid Weather Tables

• CPU View• Storage View• Bandwidth View

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

• What about including Virtual Data Tables?

Message Interface Component

• Input (incoming)• DAG • Requirements • QoS requests• Status requests

• Output (incoming)• DAG Table

(“unreduced”)• Job Table

(“unreduced”)• application• Requirements• QoS

• Input (outgoing)• Message

• Output (outgoing)• Status updatesSphinx Server

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

DAG Reducer (Decomposer?)

• Input• DAG

(“unreduced”)• Available

(existing) data

• Action• Remove DAG

nodes for which input data exists

• Add Jobs to Job table

• Output• Pruned DAG

(“unpredicted”)• Jobs

(“unpredicted”)Sphinx Server

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

Sphinx Server

Prediction Engine Component

• Input:• “unpredicted”

Job• Requirements

(if available)• CPU hours• Storage hours• Bandwidth

• Profile/History

• Output: • Estimated

Resource Usage • “unaccepted”

job

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

Sphinx Server

Admission Control Component

• Input• Entity Account

Information• “unaccepted” DAG

Request• Requirements• QoS

• Predictions of Jobs (decomposed DAG) Resource Usages

• Policy Constraints• Current schedule for

next k steps• Output

• Reject DAG Request • “Remove” DAG• “Remove” Jobs

• Accept DAG with Estimated QoS

• “Unplanned” DAG• “Unplanned” Jobs

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

Sphinx Server

Planning Component

• Job Planner (independent “work” that can be fully resolved onto resources at time t)

• Input• “unplanned” Job Information

• Requirements • Dependencies (I/O)

• Job QoS Requests• Entity Account Information• Policy Constraints• Available Data• Available Applications• Available Resource Properties• Prediction of Job Resource

Usage• Output

• QoS Deliverable• “unsent” Job(s) schedule for

the next k steps

• DAG Planner (intra-dependent “work” that can not all be fully resolved onto resources at time t)

• Input• “unplanned” DAG

Information• Requirements

• DAG QoS Requests• Current Job Schedule for next

k steps• Output

• QoS Deliverable• Modified Job Schedule for

next k steps

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

Sphinx Server

Job Submission Component (also part of Sphinx Client)

• Input• Fully planned Job:

• Job state “unsent”• Assigned gatekeeper and

jobmanager• I/O data movement locations

• Action• Construct Job DAG

• Set up environment• Import input data (if

necessary)• Publish to DMC (if necessary)• Run application• Export output data (if

necessary)• Publish to DMC (if necessary)• Cleanup

• environment• Local I/O data (if necessary)

• Submit to Condor-G/DAGMan• Job state “unfinished”

• Monitor Job statistics• Output

• Exit status (if available)• Final Job statistics (resource

usage)• CPU• Storage • Bandwidth

• Entity account update• Job state “remove”

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

Sphinx Server

Data Management Components

• “User Demanded Movement” Module

• Input• Source(s)• Destination(s)

• Output• DMC update

• “Pre-Planned Movement” Module (most important to Sphinx)

• Input• Available Data• Available Resources• Prediction of Data

Access Patterns• DAG (“reduced”)• Jobs (“unsent”)• Job(s) schedule for next

k steps• Output

• Data Movement• DMC update

• Prediction Module• Input:

• Historical Data Access Patterns

• Output:• Future Data Access

Patterns

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface

Sphinx Server

Information Gathering Module

• Resource Property Module• Input

• Grid Information Systems (such as MDS using GLUE Schema, etc)

• Output• Resource Property views

• Job Tracking Module• Input

• Job Monitoring information (such as Condor-G, etc)

• Output• Work Views

• Grid Weather Module• Input

• Grid Monitoring systems (such as GEMS, MonALISA, etc)

• Output• Grid Weather Views

• CPU• Storage• Bandwidth

• Grid Weather Prediction Module

• Input: • Grid Weather History

• Output: • Grid Weather Forcast for

next k steps

Data Warehouse

Multi-threadedControl Process

Planner

DAG Reducer

Information Gatherer

Prediction Engine

Data Management

Admission Control

Submitter

Message Interface