Sphinx Server
Sphinx Client
Data Warehouse
Submitter
Generic Grid Site
Monitoring Service
Resource
Message Interface
Current Sphinx Client/Server
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Message Interface
Prediction Engine
Sphinx Server
Sphinx Client
Data Warehouse
Submitter
Generic Grid Site
Monitoring Service
Resource
Message Interface
Sphinx Client/Server with V2 Components
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Message Interface
Sphinx Server
Data Warehouse
Stand alone Sphinx Server with light Client
Sphinx Client
Generic Grid Site
Monitoring Service
Resource
Message Interface
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
Sphinx Server
Data Warehouse• Message Tables• Policy Information Tables• Entity Accounting Tables• Work (DAG, Job, etc) Tables
• State• Unreduced• Unpredicted• Unaccepted (not admitted)• Unplanned (not yet feasible)• Unsent (assigned)• Unfinished (executing)• Remove (accounted or rejected)
• Entity• Resource Property Requirements• Dependency Requirements
• Do not start before “event”• I/O
• QoS Requests• Resource Assignment for next k steps
• Application Tables (in general, distributed…)
• Transformation Catalogue• Profiles (CPU, Disk, Bandwidth
requirements)• History (CPU, Disk, Bandwidth, actual
use)• Data Tables (in general,
distributed…)• Replica Catalogue (from RLS)• Profiles (Size, Bandwidth
requirements)• History (Access rate, etc)
• Resource Property Tables• CPU, Disk, Network Connection,
middleware configuration, etc• Grid Weather Tables
• CPU View• Storage View• Bandwidth View
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
• What about including Virtual Data Tables?
Message Interface Component
• Input (incoming)• DAG • Requirements • QoS requests• Status requests
• Output (incoming)• DAG Table
(“unreduced”)• Job Table
(“unreduced”)• application• Requirements• QoS
• Input (outgoing)• Message
• Output (outgoing)• Status updatesSphinx Server
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
DAG Reducer (Decomposer?)
• Input• DAG
(“unreduced”)• Available
(existing) data
• Action• Remove DAG
nodes for which input data exists
• Add Jobs to Job table
• Output• Pruned DAG
(“unpredicted”)• Jobs
(“unpredicted”)Sphinx Server
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
Sphinx Server
Prediction Engine Component
• Input:• “unpredicted”
Job• Requirements
(if available)• CPU hours• Storage hours• Bandwidth
• Profile/History
• Output: • Estimated
Resource Usage • “unaccepted”
job
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
Sphinx Server
Admission Control Component
• Input• Entity Account
Information• “unaccepted” DAG
Request• Requirements• QoS
• Predictions of Jobs (decomposed DAG) Resource Usages
• Policy Constraints• Current schedule for
next k steps• Output
• Reject DAG Request • “Remove” DAG• “Remove” Jobs
• Accept DAG with Estimated QoS
• “Unplanned” DAG• “Unplanned” Jobs
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
Sphinx Server
Planning Component
• Job Planner (independent “work” that can be fully resolved onto resources at time t)
• Input• “unplanned” Job Information
• Requirements • Dependencies (I/O)
• Job QoS Requests• Entity Account Information• Policy Constraints• Available Data• Available Applications• Available Resource Properties• Prediction of Job Resource
Usage• Output
• QoS Deliverable• “unsent” Job(s) schedule for
the next k steps
• DAG Planner (intra-dependent “work” that can not all be fully resolved onto resources at time t)
• Input• “unplanned” DAG
Information• Requirements
• DAG QoS Requests• Current Job Schedule for next
k steps• Output
• QoS Deliverable• Modified Job Schedule for
next k steps
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
Sphinx Server
Job Submission Component (also part of Sphinx Client)
• Input• Fully planned Job:
• Job state “unsent”• Assigned gatekeeper and
jobmanager• I/O data movement locations
• Action• Construct Job DAG
• Set up environment• Import input data (if
necessary)• Publish to DMC (if necessary)• Run application• Export output data (if
necessary)• Publish to DMC (if necessary)• Cleanup
• environment• Local I/O data (if necessary)
• Submit to Condor-G/DAGMan• Job state “unfinished”
• Monitor Job statistics• Output
• Exit status (if available)• Final Job statistics (resource
usage)• CPU• Storage • Bandwidth
• Entity account update• Job state “remove”
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
Sphinx Server
Data Management Components
• “User Demanded Movement” Module
• Input• Source(s)• Destination(s)
• Output• DMC update
• “Pre-Planned Movement” Module (most important to Sphinx)
• Input• Available Data• Available Resources• Prediction of Data
Access Patterns• DAG (“reduced”)• Jobs (“unsent”)• Job(s) schedule for next
k steps• Output
• Data Movement• DMC update
• Prediction Module• Input:
• Historical Data Access Patterns
• Output:• Future Data Access
Patterns
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface
Sphinx Server
Information Gathering Module
• Resource Property Module• Input
• Grid Information Systems (such as MDS using GLUE Schema, etc)
• Output• Resource Property views
• Job Tracking Module• Input
• Job Monitoring information (such as Condor-G, etc)
• Output• Work Views
• Grid Weather Module• Input
• Grid Monitoring systems (such as GEMS, MonALISA, etc)
• Output• Grid Weather Views
• CPU• Storage• Bandwidth
• Grid Weather Prediction Module
• Input: • Grid Weather History
• Output: • Grid Weather Forcast for
next k steps
Data Warehouse
Multi-threadedControl Process
Planner
DAG Reducer
Information Gatherer
Prediction Engine
Data Management
Admission Control
Submitter
Message Interface