Components for Scientific Computing
Randall Bramley
Computer Science, Indiana University
Center for Advanced Computation Research, California Institute of Technology
CAT Team Members
• Dennis Gannon: CAT architecture design• Randall Bramley: CAT/LSA design• Benjamin Temko: Globus wrangler• Fabian Breg: NexusRMI author• Prafulla Deuskar: parallel HPC++ components • Shridhar Diwan: HPC++ Manager • Madhusudhan Govindaraju: HPC++ framework, GRAM
server • David Stern: HPC++ framework design, LSA components • Juan Villacis: CAT design, GUI workspace, Java Manager • Andrew Whitaker: information subsystem, GUI InfoBrowser• Esra Akman, Deepa Viswanathan, Thomas Stuckey: alums
Large Sparse Linear Systems
• 105-107 or more unknowns• Irregular sparsity structure • Nonbanded, non-structured
– Not diagonally dominant
• Nonsymmetric in values– Causes convergence problems
The Good News for Linear Systems
• Sparse direct solvers readily available:– Umfpack, SuperLU, Y12M
– HPC banded solvers in Scalapack
• Preconditioned iterative solvers:– Aztec, SPLIB, BPKIT, ITPACK, ...
• Matrix manipulation packages: – Sparskit, SMMS, Metis, Chaco, ...
The Bad News for Linear Systems
• All methods are parameterized– enormous design space to navigate.
• Black arts abound– matrix reordering, scalings, filtering
• No existing theory provides a practical guide to crafting a solution strategy– experimentation required.
Goals of Linear System Analyzer (LSA)
• Develop “solution strategies” for linear systems• High-level rapid prototyping for combining sparse
matrix manipulations and solvers• Hot-wiring capabilities: grab linear systems from
running applications, return solutions/strategies dynamically
• High-performance capabilities: big systems, fast network protocols.
• We chose a distributed component architecture approach...
Classes of Components in the LSA
• I/O components: getting linear system data in and out of the framework
• Filter components: manipulating the system to have more favorable properties for a solver
• Solve components• Information components
LSA I/O Components
• Read from file: H/B or Matrix Market format• Write out solution vector to a file• Take input from URL on network• Take input from active object (running program)• Feed solution back to an active object• Extract linear system from framework to file
LSA Filter Components
• Reorder linear system: MinDeg, RCM, ND, MaxTraversal, BBD, Weighted Diag Block
• Scale system: max element, row, column, relative scalings
• Filter out elements based on magnitude• Parallel partitioning reorderings: Chaco, Metis
LSA Solver Components
• LAPACK dense solver (extracted)• LAPACK banded solver (extracted)• SuperLU sparse direct from Xiaoye Li• SPLIB preconditioned iterative solvers
– Thirteen iterative methods (only two are really useful)
– Seven preconditioners (four are of use)
– Heavily instrumented
• Aztec parallel iterative solvers• BLOCK
LSA Info Components
• BasicInfo: analysis of coefficient matrix (beefed-up version of Sparskit’s infofun)
• VizInfo: emily sparse matrix visualization• Wannabes:
– IterativeInfo: specialized analysis for selecting iterative solvers, preconditioners
– DirectInfo: estimates storage required, computational rate for sparse direct solvers
– Spectral: eigenvalue/singular value decomposition
Component Systems Definition
• Characteristics of “component”– Encapsulated software object providing specific functionality or
service, used with other components to build complete applications
– Peer-to-peer interactions vs. client-server.
– Applications built by connecting binaries - not via compilation/linking of sources
– Components interact via well-defined interfaces or ports.
– Connected as software IC’s in a framework
– Services need not be computational: visualization, object store, ...
• Components have swept desktop, business worlds– Microsoft COM, CORBA, Java Beans, Iris Explorer, Khoros, ...
Distributed Component Framework Services
• Component identification and handling– LDAP component information system
• Interface registry– Java RMI mechanism
• Connection management; exception handling– Combination of Java, HPC++, Nexus
• User interface / collaboration mechanisms– Java GUIs
• Hardware resource management– Globus system
DatabaseCATCATCompositionCompositionGUIGUI
Java Composition GUI
Visualizer
Local Java proxies to remote objectsRemote objects inHPC++, Java orwrapped MPI-Fortran
Intercomponent comm with Java RMI/HPC++remote invocation semantics implemented over Globus/Nexus
EvolverInitializer
Globus authentication and
instantiation of components
CAT Framework Architecture
Standard Metacomputing Model
• Application codes at top,
written in C, C++, Fortran, …
• HPC libraries (ScalaPack),
runtimes (MPI, Tulip)
• Infrastructure (Globus, Nexus,
network protocols)
• Hardware (Origins, SP2s,
CAVES, …)
Hardware: computers,visualization equipt
Basic infrastructuresoftware
Software libraries, runtime systems
Application codes
CAT Metacomputing Model
• Applications … not code (physical process modeling)
• Composition tools for organizing resources (CAT Java GUIs, managers)
• Resource (both SW and HW) location/management (Globus, LDAP DB’s)
• Actual resources: code, computers, visualization engines, databases, instruments
HardwareResources
SoftwareResources
Component Resource Management
Component Resource Composition Tools
Applications
CAT User Interfaces
Software componentfinder: LDAP based
Detail info aboutselected comp
Compositionwindow showingcomp connects
Common Component Architecture Group
• Existing component frameworks not suitable for HPC• Participants from all national labs, few universities• Key goal: develop component interface standards.
– Allow interchanging of components between frameworks, labs
– Mechanisms for a component to make its interfaces known
– Allow for parallel links, different network protocols.
• Meetings:– Unfunded meetings six times last year
– BOF meeting at SC98 (debutante party)
– Last meeting has implementable goal
Usage of Component Architectures
Init
DB Solve
DiscretizeComposition phase: instantiate and tie together comps
Init
DB Solve
DiscretizeRun phase: start comps executing
Mixed phase: addcomps dynamically
Visualize
A Possible CCA Framework System
entific IDL
Any CCA Compliant Framework
FrameworkServices
Component 1 Component 2
proxygenerator
Builder
Repository/Registry
GPorts Part of Gports specific to the framework
Abstract ConfigurationAPI
Repository API
Scientific IDL
(From RobArmstrong,
SNL)
Basic CCA mechanism: gPorts
• gPorts = generalized Ports– Borrows ideas from CORBA, COM, visual programming environments
like Iris Explorer, AVS, Khoros
– Current prototype based on CORBA-like user/provider model
– Uses type of COM-like IUnknown pattern
– Defined using linked interfaces.
• Source-target, start pt - end pt, source-sink, output-input, event source - listener, user-provider terms all used to describe the communicating ports
Basic CCA mechanism: gPorts
• Output ports can be multiplexed, but not input ports• gPorts directly* invoke methods on each other.• Allows
– standard data-flow model
– shared-memory or “data stays, components move” models
* Using underlying run-time system
Basic CCA mechanism: gPorts
outputGport inputGport
Source Target
Event Source“uses” remote methods
Event listener“provides” methods
Typed Channel
outputGport implements- methods defined by typed channel- addListener() used to connect target to outputGport
Examples:- outputGport is list of “global pointers” in HPC++- logic for argument marshaling and RPC- n x m collective communications as in MPI
Each Gport provides information about its types via strings for name, type:
Framework responsible for mapping from strings to internal representation of types
Basic CCA mechanism: gPorts
class gPortInfo { public gPortInfo(String portname, String porttype); public String getType(); public String getName();};
Basic CCA mechanism: gPorts
Channel types can be defined via an IDL spec:interface MyChannelType{ int row(in float, in array<int,3>); void mat_order(in int); };
Becomes in Java or in C++
Interface MyChannelType{ int row( float, array_int_3); void mat_order(int); };
class MyChannelType{ public: virtual int row(float, array_int_3)=0; virtual void mat_order(int)= 0; };
Defining input port requires implementing Channel type generated by IDL (or by hand)interface InputGPort extends GPort{};
Defining output port requires implementingChannel type
Mechanisms also will allow hierarchical compositionSee web pages for details, examples.
Basic CCA mechanism: gPorts
interface OutputGPort extends GPort{ public void addInputListener(InputGPort ); public void removeInputListener(InputGPort ); public Enumeration getInputPorts(); };
CCA and IDL
• CCA standard does not require using IDL. • CAT/LSA allows seamless interaction between Java and
HPC++ method invocations; no IDL• Groups can define own “glue” mechanisms• Must support C/C++, Java, Fortran77, Python, Perl
– Fortran 90 support allowed but not required.
• Reflection capabilities required for interfaces.– Run-time discovery of objects by frameworks
CCA Dynamic Registry
• First Level: how to register components so that frameworks can locate and instantiate them
• Second Level: how a component makes its interfaces known to the framework
• Meta-Level: how to describe the internal functionality of a component in the global marketplace
• Less of a problem in DOE applications: relatively few components and interfaces to handle.
• Will be/is a major problem for academic applications
Current CCA Status
• Scrupulous avoidance of word “standards”. • draft gPort spec
http://z.ca.sandia.gov/~cca-forum/gport-spec
• draft IDL spechttp://www.llnl.gov/CASC/babel
• Multiple test frameworks being developed– InDEPS (Sandia, Armstrong and Melius)
– CAT/LSA (IU, Gannon and Bramley)
– PAWS (LANL, Beckman and Reynders)
• Existing frameworks likely to be made CCA-compliant– POOMA, PetSC, SciRun, ISIS++, ...
Current CCA Participants
• Sandia: Rob Armstrong, Robert Clay, Bill Mason,Carl Melius • Indiana University: Dennis Gannon • CACR, Indiana: Randall Bramley • LLNL: Andy Cleary, Scott Kohn, Brent Smolinski• LANL: Pete Beckman, Pat Fasel, Bill Humphrey, Kate
Keahey• ORNL: Al Geist, Noel Nachtigal • ANL: Satish Balay, Lori Freitag, Paul Hovland, Lois Curfman
McInnes, Barry Smith • Utah: Steve Parker, Chris Johnson• Lawrence Berkeley: Brent Milne• Pacific Northwest Labs: Jarek Nieplocha
An Idiosyncratic View
• Components are the future of scientific/eng computing– desktop, business computing always leads the way
– is the right level of abstraction for object-orientation
– maps naturally to collaborating groups distributed across the globe
– integrates weird stuff: symbolic computing, vizualization, DBs
• We have to address scalability in terms of numbers of components: 105 components, 108 interface instances
• Software agents may be right tool to find, challenge/test, and connect software components