ML: A Multilevel Preconditioning Package

Copper Mountain Conference on Iterative MethodsCopper Mountain Conference on Iterative Methods

March 29-April 2, 2004

Jonathan HuRay Tuminaro

Marzio Sala

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

Outline

• Overview

• Multigrid basics

• Available user options

• Configuring & building

• Interoperability with other packages

• Example program

• Conclusions

ML Package

• Provides parallel multigrid preconditioning for linear solver methods• Current developers: Ray Tuminaro, Jonathan Hu, Marzio Sala

• Former: Charles Tong (LLNL)

• Main methods– Geometric

• Grid refinement hierarchy• 2-level FE basis function domain decomposition

– AMG (algebraic multigrid based on aggregation)• Smoothed aggregation*• Edge-element AMG for Maxwell’s equations*• n-level (smoothed) aggregation domain decomposition *• Classical AMG

• Written primarily in C– C++ interfaces to various Trilinos packages– 102,411 lines of code (as of this afternoon)– 26 example programs– Downloadable as part of Trilinos– CVS, bugzilla

MG(f, u, k){

if (k == 1) u1 = (A1)-1f1

else {

Sk(Ak, fk,uk) //pre-smooth

rk = fk – Ak uk

fk-1 = Rk-1 rk; uk-1 = 0 //restrict

MG(fk-1, uk-1, k-1)

uk = uk + Ik-1 uk-1 //interpolate & correct

Sk(Ak, fk,uk) //post-smooth }

}

Ak, Rk, Ik, Sk required on all levels

Sk: smoothers

Rk: restriction operators

Ik: interpolation operators

Recursive MG Algorithm (V Cycle)

V Cycle to solve A4u4=f4

k=4

k=1

W Cycle FMV Cycle

ML Capabilities

• MG cycling: V, W, full V, full W

• Grid Transfers– Several automatic coarse grid generators– Several automatic grid transfer operators– Coarse “grid” visualization capabilities

• Smoothers– Jacobi, Gauss-Seidel, Hiptmair, Krylov methods, sparse

approximate inverses, Chebyshev

• Variety of Serial & Parallel Direct Solvers– SuperLU, Umfpack, KLU, MUMPS, etc.

• Kernels: matrix/matrix multiply, etc.

Smoothed Aggregation

• Developed for linear elasticity (Vanek, Mandel, Brezina)

• Construct tentative prolongator tP– Aggregation: group unknowns together– Augment: interpolate null space (e.g., rigid body

modes)

• Construct final prolongator P– Smooth: P = (I – D-1A)TP

• lower energy in basis functions

5

5 5 2 2

2 7

7 72

Take null space(e.g., const.) SmoothSplit into local basis

functions

Smoothed Aggregation

Uncoupled / MIS Aggregation

• Greedy algorithm

● Global graph partitioning

– Operates on global domain

– ParMETIS

– Aggregates can span processors

Graph partitioning aggregation

● Local graph partitioning

– Processors work independently

– METIS

– 1 aggregate per processor

ML Smoother Choices

• Jacobi, Point/Block Gauss Seidel

• MLS– Based on Chebyshev polynomials of smoothed operator.– Serial: competitive with true Gauss-Seidel– Parallel: Performance is independent of # processors

• Hiptmair– Distributed relaxation smoother for Maxwell’s Eqns.– Other smoothers used within each projection step

• Aztec solvers– Krylov methods, incomplete factorizations

• Direct solution (via Amesos interface):– UMFPACK, KLU (serial)– SuperLU– MUMPS

ML and Other Packages

ML

Epetra

Accepts user data as Epetra objects

Can be wrapped as Epetra_Operator

TSF

TSF interface exists

Othermatvecs

Othersolvers

Accepts other solversand MatVecs

AmesosAmesos interface fordirect solvers

Aztecoo

Meros

Via Epetra & TSF

Configuring and Building ML

• Builds by default when you configure & build Trilinos

• By default, you get

– Epetra & Aztecoo support

– Example suite (Trilinos/packages/ml/examples)

• Some options of interest (off by default)

– MPI support

– Graph partitioning aggregation (METIS, ParMETIS)

– Direct solvers (Amesos)

– Profiling

configure --with-mpi-compilers=/usr/local/mpich/bin--with-ml_amesos \--with-libs=“-lamesos –lsuperlu”

A Small Example:ml/examples/ml_example_epetra_preconditioner.cpp

LinearSolver

ML: multi-grid pre-

cond.

AztecOO(Epetra)

ML(Teuchos)

gmres

AMG

Solution component

Example methods

Packagesused

Solve Ax=b:A: advection/diffusion operatorLinear solver: gmresPrecond.: 2-level AMG, graph-

partitioning aggregation

ml_example_epetra_preconditioner.cpp

Trilinos_Util_CrsMatrixGallery Gallery(“recirc_2d", Comm);

Gallery.Set("problem_size", 10000);

// linear system matrix & linear problem

Epetra_RowMatrix * A = Gallery.GetMatrix();

Epetra_LinearProblem * Problem = Gallery.GetLinearProblem();

// Construct outer solver object

AztecOO solver(*Problem);

// Set some solver options

solver.SetAztecOption(AZ_solver, AZ_gmres);

solver.SetAztecOption(AZ_output, 10);

solver.SetAztecOption(AZ_kspace, 160);

example (contd.) // Set up multilevel preconditioner ParameterList MLList; // parameter list for ML options

MLList.set("max levels",2); MLList.set("aggregation: type", "METIS"); // graph partitioning MLList.set("aggregation: nodes per aggregate", 16);

// set up aztecoo smoother MLList.set("smoother: type","aztec"); int options[AZ_OPTIONS_SIZE]; double params[AZ_PARAMS_SIZE]; AZ_defaults(options,params); options[AZ_precond] = AZ_dom_decomp; options[AZ_subdomain_solve] = AZ_ilut; MLList.set("smoother: aztec options", options); MLList.set("smoother: aztec params", params);

MLList.set("coarse: type","Amesos_Superludist"); MLList.set("coarse: max processes", 4);

ML_Epetra::MultiLevelPreconditioner * MLPrec = new // create preconditioner ML_Epetra::MultiLevelPreconditioner(*A, MLList, true);

solver.SetPrecOperator(MLPrec); // tell solver to use ML preconditioner solver.Iterate(500, 1e-12); // iterate at most 500 times

AMG for Common Problem Types

Trilinos_Util_CrsMatrixGallery Gallery(“laplace_3d", Comm);

Gallery.Set("problem_size", 100*100*100);

// linear system matrix & linear problem

Epetra_RowMatrix * A = Gallery.GetMatrix();

Epetra_LinearProblem * Problem = Gallery.GetLinearProblem();

// Construct outer solver object

AztecOO solver(*Problem);

solver.SetAztecOption(AZ_solver, AZ_cg);

// Set up multilevel precond. with smoothed aggr. defaults

ParameterList MLList; // parameter list for ML options

ML_Epetra::SetDefaults(“SA”,MLList);

ML_Epetra::MultiLevelPreconditioner * MLPrec = new // create preconditioner

ML_Epetra::MultiLevelPreconditioner(*A, MLList, true);

solver.SetPrecOperator(MLPrec); // tell solver to use ML preconditioner

solver.Iterate(500, 1e-12); // iterate at most 500 times

Other problem types: “DD”, “DD-ML”, “maxwell”

Collaborations

• Within Sandia– ALEGRA

• Radiation• Maxwell• Electrostatic potential

– MPSalsa (Shadid, et al.)– CEPTRE

• External users– EM3D (Lawrence Berkeley NL)– P. Arbenz (ETH Zurich)– R. Geus (PSI)– J. Fish, H. Waisman (RPI)

• Potential Users– PREMO (Sandia)

Getting Help

• Website: www.cs.sandia.gov/~tuminaro/ml• See Trilinos/packages/ml/examples• See guide:

– ML User’s Guide, ver. 3.0 (in ml/doc)• Mailing lists

– ml-users@software.sandia.gov– ml-announce@software.sandia.gov

• Bug reporting, enhancement requests via bugzilla:– http://software.sandia.gov/bugzilla

• Email us directly– jhu@sandia.gov– rstumin@sandia.gov– msala@sandia.gov

Conclusions

• ML 3.0 release this May– Part of next Trilinos release

• Existing options– Uncoupled / MIS aggregation– Smoothers– SuperLU

• New options with ML 3.0– Graph-based aggregation– “parameter list” setup– Amesos interface to direct solvers– Visualization, aggregate statistics

• Use ML with Trilinos!– ML as preconditioner is trivial– Rich set of external libraries