Embed Size (px)
Citation preview
/cog
The Globus Alliance contains the following members in alphabetical order:
The Java CoG Kit
Gregor von LaszewskiArgonne National Laboratory
University of Chicago
http://www.cogkit.orgUpdated slides will be available on the CoG Kit web site
/cog
/cog
Funding sources & Acknowledgement
The Java CoG Kit receives funding from thefollowing sponsors DOE MICS NSF NMI
Previous versions of the CoG Kit also receivedfunding from NCSA Alliance
Please, contact [email protected] in case youlike to work with us more closely.
Acknowledgement: CoG Team, Globus Team, Globus Alliance, many
others as listed on www.cogkit.org
/cog
Community
Call on the community to help us withextending and improving the CoG Kit
/cog
Outline
What is the CoG Kit? Basic definitions History of the CoG Kit CoG Kit in action Relationship to GT versions
Selected Project Components Design: Abstractions Programming with Abstractions (Task Graphs) Visual components: Portals & Applications
Conclusion
/cog
Introduction
/cog
Observation
Problem Many application developers desire to program the Grid in
familiar higher level frameworks that allow rapidprototyping.
Solution We propose to reuse a variety of commodity tools,
protocols, approaches, methodologies, while integratingGrid software based on the Globus Toolkit Easier development of advanced Grid services Easier and more rapid application development Easier deployment of Grid services Code reuse and use of component repositories Use of Web services as part of the Grids Widespread use of the Grid Use of commodity technology is not limited to the client!
/cog
Abstractions
Hypothesis: With rapidly changing technologies it may
be beneficial to have an abstraction thatcan be assisting in this technical challenge.
Solution: CoG Kit abstractions are defined for
precisely that reason.
/cog
Result: CoG Kits
CoG Kits make Grid programming simpleand new technologies are easy to integrate
We focus on a CoG Kit for Java Others are possible Python, …
Availability: Java CoG Kit since 1997
Our designs are based on experiencereaching back to the beginnings ofMeta-computing and Grid-computing
/cog
Relationship towards GT
Since GT3 CoG Kit is an essential part of GT
CoG Kit protects from an evolving standard
CoG Kit provides simple programming model
CoG Kit supports portal and GUI developers
CoG Kit is a bridge between application and Gridmiddleware developers.
CoG Kit has known to be working with GT1.0 GT2.4, GT3.0.2, GT3.2, GT3.21, SSH
(under dev.) GT3.9.x, GT4, Condor
(community) Unicore
/cog
Relationship to WS-RF Because …
(Quote: Steve Tuecke, at a GGF meeting):“WS-RF is still under development. The OASIS standardsprocess has just begun.”
COG Kit Provides investment protection while standards are
developed. Provides a more sophisticated programming model than
just services Focus on what many high end-users need You can influence the direction of the CoG Kit by
partnering with us Will work with future versions of GT, SSH, Condor
(planed), … We intend to support and integrate with upcoming new
standards
/cog
History
/cog History (cont.)
CoG Kit was selected by IBM to demonstrate Grid computing in Boards ofDirectors meeting
2001
von Laszewski joins ArgonneNov. 1996
CoG Team: LDAP browser wins Novel developers award1998
CoG Team: The experimental personal gatekeeper of the Java CoG Kit wasbeen able to be installed in less than 30 seconds on a PC including Windows,a similar Globus service installed by an experienced administrator required oneto multiple days.
2000
Term Metacomputer is introduced1992
I-Way1995
Globus Team defines OGSI / Java CoG Kit for GT2.x and GT3.0/OGSI based,includes visual components such as the CoG Kit Desktop, GridFTP interface,GRAM interface
2002
Cog Team: The Java CoG Kit experimental Infogram Service architecture wasdefined combining execution and information Services as a single Grid service.
2001
Term Java CoG Kit is introduced to include jglobus and other components ina single toolkit
1999
Term Grid is introduced1998
Globus version 1 / first release of jglobus based on concepts of protocols and services includes a high throughput fault tolerant workflow prototype
1997
von Laszewski: Graphical Meta-computing environment1994
/cog History
CoG Kit receives best research poster award at SC 20042004
CoG team rewrites the workflow component and introduces GridAnt and anew workflow engine called Karajan that contains flow and structuralcontrol (DAGs, conditions, loops). The workflow concept is expandable.Check pointing and minimal features for fault tolerance are available.
Result caching is possible based on method signatures.
2002 and2003
WSRF is defined2003
A class project shows it is possible to define PBS and LSF providers (notdistributed with the CoG Kit)
2004
Major new Java CoG Kit release.
*GT2, GT3, GT4, SSH providers* Workflow* Graphical components* New manual
2005
CoG team introduces the concept of Grid providers making it possible thatthe CoG Kit can in principal submit to GT2, GT3, GT4, or SSH. Communitydemonstrates also UNICORE provider.
2004
/cog
J a v aJ a v a
C o GC o G
K i tK i t
O G S A / O G S IO G S A / O G S I
G l o b u s T o o l k itG l o b u s T o o l k it
V e r s i o n 2V e r s i o n 2
F i l e T r a n s f e rF i l e T r a n s f e r
G r i d F T PG r i d F T P & R FT & R FT
Mi
dd
le
wa
re
Mi
dd
le
wa
re
Po
rt
al
wa
re
Po
rt
al
wa
re
Ap
pl
ic
at
io
ns
Ap
pl
ic
at
io
ns
Gr
id
Gr
id
S
er
vi
ce
s
Se
rv
ic
es
J o b S u b m i s s i onJ o b S u b m i s s i on
S e c u r i t yS e c u r i t y
W o r k fl o wW o r k fl o w
T a s k M a n a g e m entT a s k M a n a g e m ent
I n f o r m a t i o n I n f o r m a t i o n
S e r v i c e sS e r v i c e s
G l o b u s T o o l k itG l o b u s T o o l k it
V e r s i o n 3V e r s i o n 3
N P A C IN P A C I
G r i d L a bG r i d L a b
OGCEOGCE
G A D UG A D U
A c c e s s G r i dA c c e s s G r i d
C h i m e r aC h i m e r a
G R I P /G R I P / U n i c o r eU n i c o r e
P e g a s u sP e g a s u s
C l i m a t eC l i m a t e
H E PH E P
C h e m i s t r yC h e m i s t r y
G e n o m eG e n o m e
P a r a m e t e r S t u d i e sP a r a m e t e r S t u d i e s
A s t r o n o m yA s t r o n o m y
P r o d u c t i o nP r o d u c t i o n
F r a m ework sF r a m ework s
J a v aJ a v a
P y t h o nP y t h o n
C h e fC h e f
N a n oN a n o M a t e r i a l s M a t e r i a l s
C o GC o G
L a u n c h p a dL a u n c h p a d
C a c t u sC a c t u s
G r i d S p h e r eG r i d S p h e r e
X C A TX C A T
P A C IP A C I
D O ED O E C E R NC E R N
G A F 4 JG A F 4 J
N i m r o d / GN i m r o d / G
G r i d A n tG r i d A n t
P e r lP e r l
C o GC o G
K i tK i t
OGCEOGCE
Use of CoG Kits
/cog
Design
/cog
Design
Based on layered model
Flexible
Expandable
Based on Java interfaces
Abstracts protocols
Abstracts services
Provides workflow
/cog
CoG Kit is more than jglobusJava CoG Kit v4
Documentation Source Community
Unicore prov iderjglobus
abstractions(core)
Web Page
Manual
Wiki
Static pages
Gsissh/term
Cert managementgridfaces
Task management
Swing
portlets
Certificate Authority
Matlab
CoG Workflow/gridant
CoGtop /GridDesktop
CoGShell / Grid Shell
Essential part of GT3.02GT3.2, GT3.2.1GT3.9.x, GT4.0
/cog
CoG Abstraction Layer
CoG CoG CoG CoG CoG
CoG Data and Task Management Layer
CoG Gridfaces Layer
CoG CoG
CoG
GridID
E
GT2GT3OGSIclassic
GT4WS-RF Condor Unicore
Applications
SSHOthersAvakiSETI
Nanomaterials
Bio-Informatics
DisasterManagement Portals
CoG Abstraction Layer
CoG CoG CoG CoG CoG
CoG Data and Task Management Layer
CoG Gridfaces Layer
CoG CoG
CoG
GridID
E
DevelopmentSupport
CoG Abstraction Layers
/cog
Selected Project Components
/cog
Focus on Reusable APIs & Components
Abstractions Provide a simple programming model
Workflow Workflow abstraction
Portals Supporting APIs, abstractions and implementations for
portals. jglobus1.2
GSI security in Java GRAM protocal & client gridFTP protocol & client Myproxy client
Not just API’s but also their implementation
/cogFocus on Abstractions andPatterns
Abstraction above Grid Toolkits Task Model
Jobs, information query, file transfer, authentication,others
Gridfaces model Abstract views of GUIs to the Grid in different
implementations (SWING, JSP, Portlets, …)
Data Types Queues, Sets, Brokers, Schedulers. Based on Task
model
/cog
Java CoG Kit abstractions
A programming model based on a task model thatsimplifies elementary Grid patterns such as jobexecution, file transfer, and file operations.
A programming model that includes executionflows in the form of directed acyclic graphs (DAG).
The programming model is decoupeling thedefinition from the implementation, thusproviding independence from current and futureGrid versions.
Only elementary Grid patterns are considered. It makes programming the Grid simple It makes developing Grid portals more easy Focus is selected functionality
/cog
Design
ExecutableObject
Task
TaskGraph
Handlers
Events
Service
/cogDesign
ExecutableObject
Identity Status
TaskTaskGraph
SecurityContext
ServiceContact
Specification
JobSpecification
FileTransferSpecification
FileOperationSpecification
TaskHandler
*
1
1
1
1
1
1
1
TaskGraphHandler
1
1
1
*
Dependency1
1
* 1
Set Queue1
1
1
1
Service* 1
/cog
Programming with Abstractions
/cog
A simple Programming Patternpublic class COG implements StatusListener{ public void create() { … } public void submit () { … } public void statusChanged (StatusEvent e) { … } public static void main (String arg[]){
try { COG cog = new COG(); cog.create(); cog.submit(); catch (Exception e) { logger.error(“Something went wrong:”, e); }}
/cog
Executing a Simple TaskGraphTaskGraph tg = new TaskGraphImpl();
public void create () { // define tasks ….. /* Add the tasks to the TaskGraph */ tg.add(task1); tg.add(task2); tg.add(task3); tg.add(task4); tg.addDependency(task1, task2); tg.addDependency(task1, task3); tg.addDependency(task2, task4); tg.addDependency(task3, task4);}
public void submit() { TaskGraphHandler handler = new TaskGraphHandlerImpl(); try { handler.submit(tg); } catch (Exception e) { logger.error(``Some Error occured'', e); System.exit(1); }}
Task 1
Task 2
Task 4
Task 3
/cog
Create a taskTask task1 = new Task();
JobSpecification spec = new JobSpecificationImpl();spec.setExecutable(“/bin/ls”);spec.addArguments(“-la”);spec.setStdOutput(“output.txt”);
task1.setSpecification(spec);
// bind the task (late binding)
/cog
Status Monitoringpublic void statusChanged (StatusEvent event) {
Status status = event.getStatus( );
logger.debug(``Status changed to '' + status.getStatusCode());
if (status.getStatusCode( ) == Status.COMPLETED) { logger.info(``Task Done''); elsif (status.getStatusCode( ) == Status.FAILED) { logger.info(``Task Failed''); System.exit(1); }}
Users can design their ownEvent handeling logic basedon status changes
/cog
Using the Handler
try { handler.submit (cog); } catch (InvalidSecurityContextException ise) { logger.error(``Security Exception'', ise); System.exit(1); } catch (TaskSubmissionException tse) { logger.error(``TaskSubmission Exception'', tse); System.exit(1); } catch (IllegalSpecException ispe) { logger.error(``Specification Exception'', ispe); System.exit(1); } catch (InvalidServiceContactException isce){ logger.error(``Service Contact Exception'', isce); System.exit(1); }
DetailedinformationCan be retrievedif exceptions areused
/cog
Bind a Task to a ServiceService service = new ServiceImpl(Service.JOB_SUBMISSION);service.setProvider(``GT3_2_1'');
// Set Security Context – e.g. certificates and suchSecurityContext securityContext =
CoreFactory.newSecurityContext(``GT3_2_1'');securityContext.setCredentials(null); // e.g. set it to default in ./globusservice.setSecurityContext(securityContext);
// Set Contact – e.g. where to go toServiceContact serviceContact = new ServiceContactImpl(
“http://127.0.0.1:8080/ogsa/services/base/gram/ MasterForkManagedJobFactoryService”);service.setServiceContact(serviceContact);
task.setService(Service.JOB_SUBMISSION_SERVICE, service);
ServiceContact serviceContact = new ServiceContactImpl( “http://127.0.0.1:8080”);
/cog
Bind a Task to a ServiceService service = new ServiceImpl(Service.JOB_SUBMISSION);service.setProvider(``GT3_2_1'');
// Set Security Context – e.g. certificates and suchSecurityContext securityContext =
CoreFactory.newSecurityContext(``GT3_2_1'');securityContext.setCredentials(null); // e.g. set it to default in ./globusservice.setSecurityContext(securityContext);
// Set Contact – e.g. where to go to
service.setServiceContact(serviceContact);
task.setService(Service.JOB_SUBMISSION_SERVICE, service);
/cog CoG Kit Desktop
GridShell
JobIcons
MachineIcons
GridLog
NativeIcons
FileTransferGUI
/cog Portlets: OGCE.org
/cog
Contributing
You can contribute
We have a module concept allowingcomponents to be integrated in thedistribution easily
/cog
Conclusion
Programming with CoG abstractions is simple
We envision multiple programming models in CoG
We envision multiple backend services
We can support multiple protocols
We like to engage the community
Contributions: CA management, Unicore provider, gsissh
These contributions are being integrated.
