Upload
stamos
View
25
Download
1
Tags:
Embed Size (px)
DESCRIPTION
The Gateway System. This project is a collaborative effort between Northeast Parallel Architectures Center (NPAC) Ohio Supercomputer Center (OSC) Aeronautical Systems Center (ASC) MSRC. Overview. Goals. - PowerPoint PPT Presentation
Citation preview
5/24/99 Gateway Workshop
T. Haupt 1
The Gateway System
This project is a collaborative effort between • Northeast Parallel Architectures Center (NPAC)
• Ohio Supercomputer Center (OSC)
• Aeronautical Systems Center (ASC) MSRC
5/24/99 Gateway Workshop
T. Haupt 2
Overview
5/24/99 Gateway Workshop
T. Haupt 3
Goals
• To provide a problem-oriented interface to more effectively utilize HPC resources from the desktop via the Web browser.
• This “point & click” view hides the underlying complexities and details of the HPC resources and creates a seamless interface between the user’s problem description on his/her desktop system and the heterogeneous computing resources
• These HPC resources include supercomputers, mass storage systems, databases, workstation clusters, collaborative tools, and visualization servers.
5/24/99 Gateway Workshop
T. Haupt 4
Seamless Access
• Create an illusion that all resources needed to complete the user tasks are available locally.
• In particular, an authorized user can allocate the resources she needs without explicit login to the host controlling the resources.
• An analogy: NSF mounted disk or a network printer.
5/24/99 Gateway Workshop
T. Haupt 5
Services User Modules
Back-End Resources
Front-End
Back-End services comprise Tier 3.
Distributed, object-based, scalable, and reusable Web server, Object brokerand Resource Manager Middleware forms Tier 2
Three-Tier Architecture
Tier 1 is a high-level, browser-based Front End for visual programming (including selection of applications, generation of input data sets, specification of resources, post-processing and visualizations)
Towards a complete solution ...Problem description:I need to model the surface damage due to the impact
of laser to harden the material bulk. I need access to models including material bulk properties and interaction with intense electromagnetic fields.
Task description: I need 64 nodes of SP-2 at Argonne to run my MPI-based executable “a.out” you can find in “/tmp/users/haupt” on
marylin.npac.syr.edu. In addition, I need any idle workstationwith jdk1.1 installed. Make sure that the output of my a.out is
transferred to that workstation
Middle-Tier: map the user’s task description onto the resource specification; this may include resource discovery, and other services
Resource Specification
Resource Allocation: run, transfer data, run
7
PSE Example: CCM
Ken Flurchick, http://www.osc.edu/~kenf/theGateway
1. Enter the Gateway system2. Define your problem3. Identify resources (software and hardware)4. Create input file5. Run your application6. Analyze results
Target Architecture
Middle-Tier
Resource Specification
Abstract Task Specification
CTA specificknowledgedatabases
VisualAuthoring
Tools
User andGroup
Profiles
ResourceIdentificationand Access
VisualizationsCollaboration
WebFlow
Back-End Resources
Problem Solving Environment
5/24/99 Gateway Workshop
T. Haupt 9
Design Issues
• Support for a seamless access (security)
• Support for distributed, heterogeneous Back-End services (HPCC, DBMS, Internet, ...) managed independently from Gateway
• Variable pool of resources: support for discovery and dynamical incorporation into the system
• Scalable, extensible, low-maintenance Middle Tier
• Web-based, extensible, customizable, self-adjusting to varying capacities and capabilities of clients (humans, software and hardware) front end
5/24/99 Gateway Workshop
T. Haupt 10
Gateway Implementation
• Distributed, object-oriented middle tier– CORBA objects (Gateway Containers, Gateway
Modules and Gateway Services) implemented in Java. [Scalable, extensible, low-maintenance middle tier]
– Containers define the user environment.– Modules and Services serve as proxies: they accept the
user requests (Front End) and delegate them to the Back End. [Support for distributed, heterogeneous back-end services managed independently from Gateway]
Note: modules can be implemented in C++; also can be DCOM components
5/24/99 Gateway Workshop
T. Haupt 11
Gateway Implementation (2)
• Gateway operates in a keberized environment [Support for a seamless access]
– tickets are generated on the client side
– Keberos-based CORBA security service is used to manage the user sessions
– Globus GSSAPI implemented over Keberos is used for resource allocation
5/24/99 Gateway Workshop
T. Haupt 12
Gateway Implementation (3)
• Task Specification is expressed in XML– CTA independent
– Decouples implementation of the Front End and the Middle Tier
– Allows for an abstract (platform independent) task specification, and thus the Middle Tier may act as a resource broker
• Resource Specification is expressed in XML– Simplifies match-making and resource discovery
– Simplifies generating Globus RSL in-the-fly
[Support for distributed, heterogeneous Back-End services; Variable pool of resources; Scalable, extensible, low-maintenance Middle Tier]
5/24/99 Gateway Workshop
T. Haupt 13
Gateway Implementation (4)
• Component-based Front-End [extensible]
• Front-End Components (“toolbox interfaces”) are – applets (interfaces for common services)– XML pages or frames
[Web-based, extensible, customizable, self-adjusting]
• All components (Front End, Middle-Tier) are defined in XML and contain metadata (used for component mining)
5/24/99 Gateway Workshop
T. Haupt 14
Front End
5/24/99 Gateway Workshop
T. Haupt 15
CTA specific knowledge database
• Subject of Ken’s talk– requires server side support (both the middle tier and
the back-end) through well defined interfaces
– should be constructed from reusable or cloneable components
– allows for identification of software components best suited to solve the problem at hand
5/24/99 Gateway Workshop
T. Haupt 16
Visual Authoring Tools
• Allows for composition of the computational task from components (reusable modules)
• Different tools to support various programming models such as data parallel, task parallel, data flow, object oriented
• No assumption on granularity
• Metadata about components and support for archiving and mining the components
• Support for instrumentation and steering
17
Example: Data Flow
5/24/99 Gateway Workshop
T. Haupt 18
Example: DARP
5/24/99 Gateway Workshop
T. Haupt 19
User and Group Profile
• Controls the user/group environment– file access
– job monitoring
– ...
• Allows for customization– preferences
– users with disabilities
– ...
• History of actions
• Scientific notebook
5/24/99 Gateway Workshop
T. Haupt 20
Resource Identification and Access
• Computational resources– hardware, software, licenses– desktop applications
• Data– file systems, mass storage, distributed databases– Internet data repositories
• Networks
5/24/99 Gateway Workshop
T. Haupt 21
Visualizations, Collaboration, ...
• Baker/Clarke’s talk on SciVis
• Geoffrey’s talk on Collaborative Tools• support for “streaming” applications as
components• support for heterogeneous hardware
(capabilities/capacities)
5/24/99 Gateway Workshop
T. Haupt 22
Front-End infrastructure
5/24/99 Gateway Workshop
T. Haupt 23
Front-End Support
• Portal Page
• User Context
• Control Applet
• Navigator (extensible, customizable)
• PSE specific toolboxes– A placeholder for the Problem Description toolboxes
– A placeholder for the code toolbox
– Resource request toolbox – Data postprocessing toolbox
• Other (Collaboration, Visualizations, …)
5/24/99 Gateway Workshop
T. Haupt 24
Portal Page
• Provides initial access to the Gateway.
• After mutual authentication of the user and the Gateway server, creates a user context, and returns a (signed) control applet.
5/24/99 Gateway Workshop
T. Haupt 25
User Context
• Represents a Gateway session.
• The session is associated with a user (or group) profile.
• WebFlow extends the notion of the UNIX profile via the 'User Data Base' (UDB). This UDB contains information about submitted jobs, history of the users actions, and other user state information. The user context may also contain application/front-end specific information.
5/24/99 Gateway Workshop
T. Haupt 26
5/24/99 Gateway Workshop
T. Haupt 27
Control Applet• The control applet is responsible for maintaining the
session, and direct communication with the middle-tier. • Direct communication is the most efficient, but since it is
buried into an applet, this mechanism is not readily customizable.
• The generic services, such as file service (upload, download, edit, copy, move, delete) and job services (show current jobs/show queues/kill jobs) will be supported this way. [combination of the user context and a query]
• The Gateway will also support a non-direct communication with the middle-tier through servelts.
5/24/99 Gateway Workshop
T. Haupt 28
Screen Dump of the Control Applet
5/24/99 Gateway Workshop
T. Haupt 29
Navigator
• The navigator allows the user to select and customize toolboxes.
• Embedded in a separate frame, it consists of menus, buttons, links, etc, derived from an XML document.
• The navigator is a hierarchical, extensible and customizable.
5/24/99 Gateway Workshop
T. Haupt 30
5/24/99 Gateway Workshop
T. Haupt 31
Problem description toolboxes
• The problem description is application specific, and the Gateway only provides a general framework for creating a PSE.
• The most important part is the specification of what services (middle and back tier) are needed, what is their API, and how to add new services.
• Example services: access to databases, XML parsing, generating HTML in-the-fly, file services.
5/24/99 Gateway Workshop
T. Haupt 32
Code toolboxes
• The end user see it as a mapping between the problem description and software to be used to solve the problem. Actually, it identifies WebFlow modules and their parameters to be used to construct the application (see resource request toolbox below).
• The module parameters may include input files, and if necessary, the input files are generated at this stage (using this or a separate toolbox). In addition, some parameters will be constructed from information stored in data bases, including UDB, and other sources.
5/24/99 Gateway Workshop
T. Haupt 33
Resource Request Toolbox
• The front-end activities result in an abstract task specification.
• Abstract in the sense that the user may not know nor care what actual resources are used.
• The task is composed of independently developed modules and services following different programming models.
5/24/99 Gateway Workshop
T. Haupt 34
Other toolboxes
• Visualizations
• Collaboration
• Scientific notebook
• ...
5/24/99 Gateway Workshop
T. Haupt 35
Middle-Tier
User 1 User 2
Application 1
Application 2
App 2App 1
WebFlow Server
WebFlow server is given by a hierarchy of containers
and components
WebFlow server hosts users and services
Each user maintainsa number of applications
composed of custom modules
and common services
WebFlow Services
5/24/99 Gateway Workshop
T. Haupt 37
CORBA Based Middle-Tier
Mesh of WebFlow Serversimplemented as CORBA objects
that manage and coordinate
distributed computation.
Front End
GatekeeperAuthenticationAuthorization
T. Haupt 38
WebFlow Context HierarchyMaster Server (Gatekeeper)
Slave Server
Slave Server
User Context
Application Context
Module
Slave Server Proxy
Services User Modules
Browserbased
Front-End
Middle-Tier modulesserve as proxies ofBack-End Services
Browserbased
Front-End
User Space Definition and Task Specification
Metacomputing Services
Back-End Resources
5/24/99 Gateway Workshop
T. Haupt 40
Back End
5/24/99 Gateway Workshop
T. Haupt 41
Back End Services
• Access to HPCC (via Globus)
• Access to distributed databases (via JDBC)
• Access to mass storage
• Access to the Internet resources
• Access to desktop application and local data
• Access to code repositories
5/24/99 Gateway Workshop
T. Haupt 42
WebFlow over Globus
• In order to run WebFlow over Globus there must be at least one WebFlow node capable of executing Globus commands, such as globusrun
• Jobs that require computational power of massively parallel computers are directed to the Globus domain, while other jobs can be launched on much more modest platforms, such as the user’s desktop or even a
laptop running Windows NT.
Bridge between WebFlow and Globus
5/24/99 Gateway Workshop
T. Haupt 43
How to add new Back-End hardware resources
• Computational engines– install Globus
– MDS (The Alliance LDAP server) will contain all relevant info, including contact address
– we need “private” directory, with entries in XML(learn from MDS, CONDOR’s class-add, NPAC’s …
– access control provided by Keberos (cross MSRC)
• Databases– create a new user: Gateway
5/24/99 Gateway Workshop
T. Haupt 44
Gateway Security
SECIOP
Security Model (Keberos)
Front End Applet
SECIOP
authentication& authorization
Gatekeeper
delegation
HPCC resources
GSSAPIGSSAPI
Layer 1: secure Web
Layer 2: secure CORBA
Layer 3: Secure access to resources
Policies defined by resource owners
5/24/99 Gateway Workshop
T. Haupt 46
Building Gateway Components
47
Middle-Tier is given by a mesh of WebFlow Servers that manage
and coordinate distributed computation
.
• Gateway applications are composed of independent reusable modules• Modules are written by module developers who have only limited knowledge of the system on which the modules will run.• The WebFlow system hides module management and coordination functions
5/24/99 Gateway Workshop
T. Haupt 48
How to develop a Gateway component(or a toolbox)
• Back-end service
• Middle-tier proxy
• Front-end controls
5/24/99 Gateway Workshop
T. Haupt 49
What does it take to convert a legacy (high performance) application into a
Gateway Back-End service?
• Nothing!– it is the only way we can support commercial
codes such as Gaussian
• A middle-tier proxy will submit your job for you
5/24/99 Gateway Workshop
T. Haupt 50
How the Back-End interacts with the rest of the system?
• Often, your job do not need to interact.– Using GRAM and GASS you stage data and
executable, submit the job and retrieve output.
– Using DUROC you can coallocate resources and run MPI-based parallel/distributed codes. The messages between nodes are sent outside Gateway control or support.
– HPF runtime will distribute your job and facilitate interprocess communication.
5/24/99 Gateway Workshop
T. Haupt 51
Implementing Back-End Services
• If you need to interact– Using a separate module, you may move files between
nodes while your jobs are executing– Your job may be a server (e.g., database, GRAM)
[if socket listener - be careful about security!]
– Your job my be a CORBA client (Java, C++)
– ...
5/24/99 Gateway Workshop
T. Haupt 52
What does it take to develop a
Gateway module (a proxy)? • Many come as a standard Gateway modules
• User’s modules– Are CORBA objects
• Define IDL (as an XML document)• Compile IDL (in the tie mode)
• Implement the functionality of the module
• Implement events
• Develop Front-End controls that invoke methods of the module
Example of IDL definition#include “..\BC.idl”module WebFlow { module lms{ interface runEdys:BeanContextChild { void run(); void receiveData(); void setParameter(in string p); }; interface runCasc2d:BeanContextChild{ void run(); void runAgain(); }; interface DoneEvent{ Object getSource(); }; };};
We will create 3 CORBA objects
* two modules: - runEdys - runCasc2d * one event - DoneEvent
They will be added to packageWebFlow.lms
5/24/99 Gateway Workshop
T. Haupt 54
Module functionality
• May act as a client for a back-end service such as Globus GRAM or a database
• May invoke other Gateway Middle Tier services such as File Services or Resource Discovery
• May implement the desired functionality internally (say, in Java) [thus not a proxy]
• May interact with other modules and the Front End through events
5/24/99 Gateway Workshop
T. Haupt 55
Event binding
addEventListenerrmEventListenerfireEvent(E,M)
method M
Event Source Event TargetAdapter
Event
ORB
binding table
DII DSI
5/24/99 Gateway Workshop
T. Haupt 56
Front-End controls
• Details in Ken’s talk
• How to control a Middle-Tier module?– Through an applet that serves as a CORBA
client, and invokes methods of the module
• How to build an application from modules?– Generate an XML document - task specification
and invoke server-side XML parser which will generate CORBA client code
5/24/99 Gateway Workshop
T. Haupt 57
Controlling a moduleApplet
ModuleControls
Proxy Module
ModuleActionButton1ActionButton2
….
IIOP
Another complication:Java sandbox!
5/24/99 Gateway Workshop
T. Haupt 58
Selecting a Predefined Task
Visual Authoring Tools
XMLA visual representation
is converted into a XMLdocument
XMLservice
WebServer
save
parse
ApplContext
Generates Java code to add modules to ApplContextPublishes IOR
Front-End Applet
Middle-Tier