Challenges in Performance Challenges in Performance Evaluation and Improvement Evaluation and Improvement of Scientific Codesof Scientific Codes
Boyana NorrisArgonne National Laboratoryhttp://www.mcs.anl.gov/~norris
Ivana VeljkovicPennsylvania State University
SIAM CSE 2February 13, 2005
OutlineOutline
Performance evaluation challenges
Component-based approach
Motivating example: adaptive linear system
solution
A component infrastructure for performance
monitoring and adaptation of applications
Summary and future work
SIAM CSE 3February 13, 2005
AcknowledgmentsAcknowledgments
Ivana Veljkovic, Padma Raghavan (Penn State) Sanjukta Bhowmick (ANL/Columbia) Lois Curfman McInnes (ANL) TAU developers (U. Oregon) PERC members Sponsor: DOE and NSF
SIAM CSE 4February 13, 2005
Challenges in performance evaluationChallenges in performance evaluation
+ Many tools for performance data gathering and analysis PAPI, TAU, SvPablo, Kojak, … Various interfaces, levels of automation, and approaches to
information presentation User’s point of view
- What do the different tools do? Which is most appropriate for a given application?
- (How) can multiple tools be used in concert?- I have tons of performance data, now what? - What automatic tuning tools are available, what exactly do they
do?- How hard is it to install/learn/use tool X?- Is instrumented code portable? What’s the overhead of
instrumentation? How does code evolution affect the performance analysis process?
SIAM CSE 5February 13, 2005
Incomplete list of toolsIncomplete list of tools Source instrumentation: TAU/PDT, KOJAK (MPI/OpenMP),
SvPablo, Performance Assertions, … Binary instrumentation: HPCToolkit, Paradyn, DyninstAPI,
… Performance monitoring: MetaSim Tracer (memory), PAPI,
HPCToolkit, Sigma++ (memory), DPOMP (OpenMP), mpiP, gprof, psrun, …
Modeling/analysis/prediction: MetaSim Convolver (memory), DIMEMAS(network), SvPablo (scalability), Paradyn, Sigma++, …
Source/binary optimization: Automated Empirical Optimization of Software (ATLAS), OSKI, ROSE
Runtime adaptation: ActiveHarmony, SALSA
SIAM CSE 6February 13, 2005
Incomplete list of toolsIncomplete list of tools Source instrumentation: TAU/PDT, KOJAK (MPI/OpenMP),
SvPablo, Performance Assertions, … Binary instrumentation: HPCToolkit, Paradyn, DyninstAPI,
… Performance monitoring: MetaSim Tracer (memory), PAPI,
HPCToolkit, Sigma++ (memory), DPOMP (OpenMP), mpiP, gprof, psrun, …
Modeling/analysis/prediction: MetaSim Convolver (memory), DIMEMAS(network), SvPablo (scalability), Paradyn, Sigma++, …
Source/binary optimization: Automated Empirical Optimization of Software (ATLAS), OSKI, ROSE
Runtime adaptation: ActiveHarmony, SALSA
SIAM CSE 7February 13, 2005
Challenges (where is the complexity?)Challenges (where is the complexity?) More effective use integration Tool developer’s perspective
Overhead of initially implementing one-to-one interoperabilty Managing dependencies on other tools Maintaining interoperabilty as different tools evolve
Individual Scientist Perspective Learning curve for performance tools less time to focus on own
research (modeling, physics, mathematics) Potentially significant time investment needed to find out
whether/how using someone else’s tool would improve performance tend to do own hand-coded optimizations (time-consuming, non-reusable)
Lack of tools that automate (at least partially) algorithm discovery, assembly, configuration, and enable runtime adaptivity
SIAM CSE 8February 13, 2005
What can be doneWhat can be done
How to manage complexity? Provide Performance tools that are truly interoperable Uniform easy access to tools Component implementations of software, esp. supporting
numerical codes, such as linear algebra algorithms New algorithms (e.g., interactive/dynamic techniques, algorithm
composition)
Implementation approach: components, both for tools and the application software
SIAM CSE 9February 13, 2005
What is being doneWhat is being done
No “integrated” environment for performance monitoring, analysis, and optimization
Most past efforts One-to-one tool interoperability
More recently OSPAT (initial meeting at SC’04), focus on common
data representation and interfaces Tool-independent performance databases: PerfDMF Eclipse parallel tools project (LANL) …
SIAM CSE 10February 13, 2005
OSPATOSPAT The following areas were recommended for OSPAT to
investigate: A common instrumentation API for source level, compiler level,
library level, binary instrumentation A common probe interface for routine entry and exit events A common profile database schema An API to walk the callstack and examine the heap memory A common API for thread creation and fork interface Visualization components for drawing histograms and
hierarchical displays typically used by performance tools
SIAM CSE 11February 13, 2005
ComponentsComponents Working definition: a component is a piece of software that
can be composed with other components within a framework; composition can be either static (at link time) or dynamic (at run time) “plug-and-play” model for building applications For more info: C. Szyperski, Component Software: Beyond Object-
Oriented Programming, ACM Press, New York, 1998
Components enable Tool interoperability Automation of performance instrumentation/monitoring Application adaptivity (automated or user-guided)
SIAM CSE 12February 13, 2005
Example: component infrastructure for Example: component infrastructure for multimethod linear solversmultimethod linear solvers Goal: provide a framework for
Performance monitoring of numerical components Dynamic adaptativity, based on:
Off-line analyses of past performance information Online analysis of current execution performance information
Motivating application examples: Driven cavity flow [Coffey et al, 2003], nonlinear PDE solution FUN3D – incompressible and compressible Euler equations
Prior work in multimethod linear solvers McInnes et al, ’03, Bhowmick et al,’03 and ’05, Norris at al. ’05.
SIAM CSE 13February 13, 2005
Example: driven cavity flowExample: driven cavity flow
Linear solver: GMRES(30), vary only fill level of ILU preconditioner Adaptive heuristic based on:
Previous linear solution convergence rate, nonlinear solution convergence rate, rate of increase of linear solution iterations
96x96 mesh, Grashof = 105, lid velocity = 100 Intel P4 Xeon, dual 2.2 GHz, 4GB RAM
SIAM CSE 14February 13, 2005
Example: Compressible PETSc-FUN3DExample: Compressible PETSc-FUN3D
Finite volume discretization, variable order Roe scheme on a tetrahedral, vertex-centered mesh
Initial discretization: first-order scheme; switch to second-order after shock position has settled down
Large sparse linear system solution takes approximately 72% of overall solution time Original FUN3D developer: W.K. Anderson et al., NASA Langley
Image: Dinesh Kaushik
SIAM CSE 15February 13, 2005
PETSc-FUN3d, cont.PETSc-FUN3d, cont.
A3: Nonsequence-based adaptive strategy based on polynomial interpolation [Bhowmick et al., ’05]
A3 vs base method time: ~1% slowdown - 32% improvement Hand-tuned adaptive vs base method time: 7% - 42% improvement
SIAM CSE 16February 13, 2005
Component architectureComponent architecture
PerfDMFPerfDMF
Metadata extractorMetadata extractor CheckpointCheckpoint
Runtime DBRuntime DB
TAUTAU
ExperimentExperiment
MonitorMonitor
Off-line analysisOff-line analysis
insertextract
start, stop, trigger
checkpointadapt request
adapt: algorithm, parameters
extract
extract query
SIAM CSE 17February 13, 2005
Future workFuture work Integration of ongoing efforts in
Performance tools: common interfaces and data represenation (leverage OSPAT, PerfDMF, TAU performance interfaces, and similar efforts)
Numerical components: emerging common interfaces (e.g., TOPS solver interfaces) increase choice of solution method automated composition and adaptation strategies
Long term Is a more organized (but not too restrictive)
environment for scientific software lifecycle development possible/desirable?
SIAM CSE 18February 13, 2005
Ext. dependencies,Version control
Configure, make,…
Performance tools
Job management,Results
DebuggingDebugging
Typical application development “cycle”Typical application development “cycle”
Implementation
Implementation
ProductionExecution
ProductionExecution
Compilation, Linking
Compilation, Linking
DeploymentDeployment
TestingTesting
DesignDesign
Performance evaluation
Performance evaluation
SIAM CSE 19February 13, 2005
Future workFuture work
Beyond components Work flow Reproducible results – associate all necessary
information for reproducing particular application instance
Ontology of tools and tools to guide selection and use
SIAM CSE 20February 13, 2005
SummarySummary No shortage of performance evaluation, analysis, and
optimization technology (and new capabilities are continuously added)
Little shared infrastructure, limiting the utility of performance technology in scientific computing
Components, both in performance tools, and numerical software can be used to manage complexity and enable better performance through dynamic adaptation or multimethod solvers
A life-cycle environment may be the best long-term solution
Some relevant sites: http://www.mcs.anl.gov/~norris http://perc.nersc.gov (performance tools) http://cca-forum.org (component specification)