The Open Grid Computing Environments ProjectMarlon PierceCommunity Grids LaboratoryIndiana University
AcknowledgementsFunding from NSF NMI (2003-2007) and OCI SDCI (2007-2010).Current participantsIndiana University (Pierce, Gannon)RENCI (Kandaswamy)RIT(von Laszewski)SDSC (Wilkins-Diehr)SDSU (Thomas, Edwards)TACC (Dahan)
OutlineWeb Portals and Science GatewaysOGCE effortsOGCE Portal SoftwarePortal toolsJava COG, GTLABOGCE Gateway ServicesGFAC, GPIRSoftware Engineering IssuesWhat is next?
OGCE GoalsTo provide easily installable, well-tested software for building Web client and service components that constitute a Grid Computing Environment.Science Web Portal --> GCE --> Science GatewayTo support developing groups through training, outreach, and divine intervention.Gateways have many needs that cant be solved by downloadable software alone.
What Is a Web Portal?Aggregate content from multiple sources into a single display.Typically consume RSS/Atom news feeds.More powerful versions these days support Flickr, calendars, games, etc.Gadgets, widgets Examples: iGoogle, Netvibes, My Yahoo!
Science Portals and GatewaysScience portals resemble standard portals, but must alsoSupport access to computing and storage resources.Allow users remote, Unix-like access to these resources.Provide access to science applications and data sets.So security is crucial.And we must provide value added services as well as user interfaces.
A Comprehensive Gateway Architecture
Components for Science PortalsOGCE is founded on the principal that portals should be built out of reusable parts.Key standard in our first phase: the JSR 168 portlet specification.Portlets can run in multiple containersuPortal, Sakai, GridSphere, LifeRay, etc.Allows us to build Grid specific components and deploy along side other goodies: Sakai collaboration tools, contributed portlets, etc.
OGCE Portal Software
OGCE GPIR portlet can interoperate with TeraGrid and your own GPIR services.
Manage TeraGrid MyProxy credentials with the OGCE ProxyManager portlets.
OGCE file management client portlets interact with TeraGrid GridFTP servers.
General purpose batch and interactive job submission to GRAM, WS-GRAM is supported.
Dashboard Portlet*The dashboard portlet allows users to track jobs on the selected resource. The user can view either his own set of jobs or get information on all submitted jobs.
Queue forecasting portlets work with the NWS QBETS to predict wait times and deadlines.
PURSe portlets manage user requests for portal accounts and Grid credentials.
Condor and Condor-G
OGCE IFrame Portlet can be used to integrate external sites.
Building Your Own Grid Portlets
Coding PortletsPortlets are just servlet-like Java classes.Basic API key methods:doView(), processAction().These are coupled to JSP pages (typically) through tag libraries and request dispatchers.OGCE supports Velocity portletsSo we must provide the coding logic for processAction().COG abstraction layers provide this.
CoG Abstraction Layers GT2GT3(X)GT4WS-RFCondorUnicoreApplications
TaskTaskHandlerServiceTaskSpecificationSecurityContextServiceContactThe class diagram is thesame for all grid tasks (running jobs, modifying files, moving data).Classes also abstract toolkit provider differences. You set these as parameters: GT2, GT4, etc.
Task and SpecificationTask task=new TaskImpl(mytask,Task.JOB_SUBMISSION);task.setProvider(GT2);JobSpecification spec=new JobSpecificationImpl();spec.setExecutable(rm);spec.setBatchJob(true);spec.setArguments(-r);task.setSpecification(spec);
Service and Security ContextService service=new ServiceImpl(Service.JOB_SUBMISSION);service.setProvider(GT2);SecurityContext securityContext=CoreFactory.newSecurityContext(GT2);//Use cred object from ProxyManagersecurityContext.setCredentials(cred);service.setSecurityContext((SecurityContext)securityContext);
Service Contact and SubmitServiceContact serviceContact=new ServiceContact(myhost.myorg.org);service.setServiceContact(serviceContact);task.setService(Service.JOB_SUBMISSION_SERVICE,service);TaskHandler handler=new GenericTaskHandler();handler.submit(task);
Coupling CoG TasksThe COG abstractions also simplify creating coupled tasks.Tasks can be assembled into task graphs with dependencies.Do Task B after successful Task AGraphs can be nested.
Problems with Portlet DevelopmentGrid portlets typically wrap each single Grid capability in a separate portlet Problem is that Grid portlets need to combine these operationsPortlets are entire web applications, so we need a component model for portlets: reusable portlet partsEven with the COG Abstraction Layer, we must still do a lot of coding to biuld new applications. To address these problems we have adopted Java Server FacesProvides several nice Model-View-Controller featuresJSF provides an extensible framework (tag libraries) for making reusable components.Apache JSF portlet bridge allows you to convert standalone JSF applications (development phase) into portlets (deployment phase).
Grid Tag Libraries and Beans (GTLAB)GTLAB provides common components for building portlets using tags and reusable parts.The goal of GTLAB to simplify Grid portlet developmentEnable rapid developmentGTLAB capabilities include Grid operations with XML based tags within Java Server Faces (JSF) framework.Grid tag libraries are built using JSF custom component development techniquesGrid tags are interfaces to backing Grid beansEnd users pass values to Grid beans by using tag attributes.We build on Java CoG 4s abstraction layer.Each backing Grid bean has equal capability with a portlet application in case of Grid portlet approach.
* Grid tags are associated with Grid services via Grid beans Grid Beans wrap the Java COG Kit (version 4) We show an example JSF page section below. This allows you to develop new Grid portlets with no additional Java code.
How to prepare application pagesDevelopers embed Grid tags snippet into JSF pageThese components are non-visual and are not displayed in HTML.Resource bean provides bridging with form inputs and GTLAB framework. Dynamic values to Grid tag attributes are provided by Resource bean.Only visual component is tag that is associated with action method of GTLAB.*
GTLAB Dashboard PortletExample
- Tracking and Managing Jobs GTLAB manages lifecycles of jobs and monitor their status.Grid operations are usually batch processesWe provide callback mechanism to follow up the jobsGTLAB creates handlers for jobs and persistently stores them.GTLAB handlers manages the job events such as stop, cancel or resuming the running jobs.GTLAB provides archive for job metadata and allows managing the archiveHandler tag helps to organize users job repository
OGCE Gateway Services
*Web Services in Scientific Communities (G. Kandaswamy)Web services are used to wrap scientific applications toDescribe, publish, discover and consume scientific applications in a standard wayCompose complex workflows from scientific applicationsRun and monitor complex workflows on distributed resourcesSuch web services that wrap scientific applications are called application services
*Host1Host2SOAP RequestSOAP ResponseCommand-line ArgumentsOutput ResultsA Simple Application Service
*Things Are Usually More Complicated
The ProblemApplication services may not be available during a workflow executionUnreliable resources (software, computers, networks)Heavy load on serviceDoes not meet QoS or security requirements of clientWorkflows cannot complete unless all services are available
*GFAC SolutionA Generic Application FactoryA persistent web service that knows how to create instances of any application serviceUse a Generic Application Factory to create instances of application services on-demand from workflows
*ImplementationThe Generic Application Factory (GFac)The Generic Service Toolkit: A toolkit that wraps any command-line application as an application serviceWithout writing any web service codeWithout modifying the application in any significant way
*Creating an Application Service (1/2)Write ServiceMap document to describe your serviceWrite Application Deployment Description document to describe a deployment of your applicationUpload the above two documents to a Registry service
*Creating an Application Service (2/2)5. Register capabilities5. Register WSDL3. Create service1. Create service requestPortal2. Get ServiceMap & Host Description4. Configure serviceHost1Host2Service Provider
*Invoking an Application ServicePortal4. Run application3. Return user interface4. Invoke Service7. Return results2. Access service5. Get Application Deployment Description and Host Description6. Send notificationsHost2Host3User
Software Engineering Issues
OGCE Code RepositoryWe use SourceForge, SVNhttp://sourceforge.net/projects/ogceOther SourceForge tools are useful.Replaced old OGCE bugzilla with SF bugzilla recently after we were attacked by robots.
Portal Build SystemThe portal download gives you everything you need to get started except Java.Includes Tomcat, GridSphere, Ant, and Maven.Assume you have a Grid somewhere.Build system (recently revised) is designed to build everything in one comman