Upload
ronia
View
37
Download
0
Embed Size (px)
DESCRIPTION
Condor-G: An Update. Outline. What is Condor-G Past Present Future. What Is Condor-G. Use Condor to run jobs on the Grid Uses Globus Toolkit GRAM (submit a remote job) GASS (transfer job’s files) Two components Globus Universe GlideIn. Globus Universe. Run a job on a Grid resource - PowerPoint PPT Presentation
Citation preview
Jaime FreyComputer Sciences DepartmentUniversity of Wisconsin-Madison
[email protected]://www.cs.wisc.edu/condor
Condor-G:An Update
www.cs.wisc.edu/condor
Outline
› What is Condor-G
› Past
› Present
› Future
www.cs.wisc.edu/condor
What Is Condor-G
› Use Condor to run jobs on the Grid› Uses Globus Toolkit
GRAM (submit a remote job) GASS (transfer job’s files)
› Two components Globus Universe GlideIn
www.cs.wisc.edu/condor
Globus Universe
› Run a job on a Grid resource
› Features Job management Fault tolerance Credential management
› Disadvantages No remote syscalls, checkpoint/migration, or
dynamic resource selection
www.cs.wisc.edu/condor
How It Works
ScheddSchedd
LSFLSF
Condor-G Grid Resource
www.cs.wisc.edu/condor
How It Works
ScheddSchedd
LSFLSF
Condor-G Grid Resource
600 Globusjobs
www.cs.wisc.edu/condor
How It Works
ScheddSchedd
LSFLSF
Condor-G Grid Resource
GridManagerGridManager
600 Globusjobs
www.cs.wisc.edu/condor
How It Works
ScheddSchedd JobManagerJobManager
LSFLSF
Condor-G Grid Resource
GridManagerGridManager
600 Globusjobs
www.cs.wisc.edu/condor
How It Works
ScheddSchedd JobManagerJobManager
LSFLSF
User JobUser Job
Condor-G Grid Resource
GridManagerGridManager
600 Globusjobs
www.cs.wisc.edu/condor
GlideIn
› Create your own personal Condor pool from temporarily-acquired Grid resources
› Brings the full power of Condor to the Grid
› Run a Condor startd on a Grid resource
› Startd reports back to your machine and runs Vanilla and Standard Universe jobs
www.cs.wisc.edu/condor
How It Works
ScheddSchedd
LSFLSF
CollectorCollector
Condor-G
600 Condorjobs
Grid Resource
www.cs.wisc.edu/condor
How It Works
ScheddSchedd
LSFLSF
CollectorCollector
600 Condorjobs
glide-ins
Condor-G Grid Resource
www.cs.wisc.edu/condor
How It Works
ScheddSchedd
LSFLSF
CollectorCollector
GridManagerGridManager
600 Condorjobs
glide-ins
Condor-G Grid Resource
www.cs.wisc.edu/condor
How It Works
ScheddSchedd JobManagerJobManager
LSFLSF
CollectorCollector
GridManagerGridManager
600 Condorjobs
glide-ins
Condor-G Grid Resource
www.cs.wisc.edu/condor
How It Works
ScheddSchedd JobManagerJobManager
LSFLSF
StartdStartd
CollectorCollector
GridManagerGridManager
600 Condorjobs
glide-ins
Condor-G Grid Resource
www.cs.wisc.edu/condor
How It Works
ScheddSchedd JobManagerJobManager
LSFLSF
StartdStartd
CollectorCollector
GridManagerGridManager
600 Condorjobs
glide-ins
Condor-G Grid Resource
www.cs.wisc.edu/condor
How It Works
ScheddSchedd JobManagerJobManager
LSFLSF
User JobUser Job
StartdStartd
CollectorCollector
Grid Resource
GridManagerGridManager
600 Condorjobs
glide-ins
Condor-G
www.cs.wisc.edu/condor
Globus Grid
PBS LSF
Condor
Condor-G
www.cs.wisc.edu/condor
Globus Grid
PBS LSF
Condor
600 Condorjobs
Condor-G
www.cs.wisc.edu/condor
Condor-G
Globus Grid
PBS LSF
Condor
600 Condorjobs
www.cs.wisc.edu/condor
Condor-G
Globus Grid
PBS LSF
Condor glide-ins
600 Condorjobs
www.cs.wisc.edu/condor
Condor-G
Globus Grid
PBS LSF
Condor glide-ins
600 Condorjobs
www.cs.wisc.edu/condor
Condor-G
Globus Grid
PBS LSF
Condor glide-ins
600 Condorjobs
www.cs.wisc.edu/condor
Condor-G
Globus Grid
PBS LSF
Condor glide-ins
600 Condorjobs
www.cs.wisc.edu/condor
Past
› GridManager daemon Runs Grid jobs using GRAM protocol Stages executable and standard I/O using
GASS protocol
› Globus GRAM 1.5 We added fault-tolerance to the GRAM
protocol Changes included in Globus Toolkit 2.0
release
www.cs.wisc.edu/condor
Present
› Updated Condor-G to Globus Toolkit 2.0
› Enhanced GridManager
› GAHP
www.cs.wisc.edu/condor
Enhanced GridManager
› Put problem jobs on hold
› No more stuck jobs
› Increase concurrency with GAHP
› Almost ready
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Single-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Globus Application Helper Protocol (GAHP)› Condor is non-threaded
› Want to use multi-threaded libraries Increased concurrency
› Put libraries in external helper process
› Simple interface over pipes/sockets
www.cs.wisc.edu/condor
Multi-Threaded Execution with GAHP
GridManager Grid Resource
Grid Resource
Grid Resource
Grid Resource
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution with GAHP
GridManager
GAHP Client
Grid Resource
Grid Resource
Grid Resource
Grid Resource
GAHP Server
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution with GAHP
GridManager
GAHP Client
Grid Resource
Grid Resource
Grid Resource
Grid Resource
GAHP Server
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution with GAHP
GridManager
GAHP Client
Grid Resource
Grid Resource
Grid Resource
Grid Resource
GAHP Server
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution with GAHP
GridManager
GAHP Client
Grid Resource
Grid Resource
Grid Resource
Grid Resource
GAHP Server
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution with GAHP
GridManager
GAHP Client
Grid Resource
Grid Resource
Grid Resource
Grid Resource
GAHP Server
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution with GAHP
GridManager
GAHP Client
Grid Resource
Grid Resource
Grid Resource
Grid Resource
GAHP Server
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Multi-Threaded Execution with GAHP
GridManager
GAHP Client
Grid Resource
Grid Resource
Grid Resource
Grid Resource
GAHP Server
Job 1
Job 2
Job 3
Job 4
www.cs.wisc.edu/condor
Future
› GRAM 1.6
› Condor-G on Windows
› Condor-G Grid service
www.cs.wisc.edu/condor
Globus GRAM 1.6
› Working with Globus team to add additional features to GRAM protocol Credential refresh File staging Scheduler-specific options
www.cs.wisc.edu/condor
Condor-G for Windows
› Condor Windows implementation available
› GRAM and GASS APIs No C implementation for Windows (yet) Java implementation (Java CoG)
› Condor-G Windows version possible by writing GAHP
server in Java
www.cs.wisc.edu/condor
Condor-G Grid Service
› Reliable job submission service for higher-lever applications
› Open Grid Services Architecture (OGSA)
› SOAP, WSDL, WS-Inspection› Implement Grid service interface for
Condor-G (and Condor in general)
www.cs.wisc.edu/condor
Thank You
› Condor-G demo on Wednesday 3351 CS
› Questions? Talk to me E-mail [email protected]