1

High Performance andGrid Computing

Applications with theCactus Framework

Gabrielle Allen

Department of Computer ScienceCenter for Computation & Technology

Louisiana State University

HPCC Program GrandChallenges (1995)

• Aerospace

• Environmental Modeling

• Molecular Biology

• Product Design

• Space Science: Black HoleBinaries, GravitationalWaves, Galaxy Formation

• Quantum Chromodynamics

• Oil Reservoir Modeling

• Fusion

• Computational Chemistry

• Global Climatehttp://www.hpcc.gov/pubs/blue95/

2

Cactus Frameworkhttp://www.cactuscode.org

• Programming framework andtoolkits for high performancemulti-dimensional simulations

• Parallel: Scales to thousands ofprocessors

• Open Source

• Modular, extremely portable

• Targeted at collaborativedevelopment

• Designed for scientists andengineers with Grand Challengeproblems

• Incorporate and drive new andcutting edge technologies

What is Difficult About HPC?(Where Does Cactus Help?)

• Many different architectures & operatingsystems, changing very rapidly

• Must worry about many things:– Single processor performance, caches, etc

– Different languages, operating systems

– Parallelism, I/O, Visualization

– Batch systems, file systems, allocations

• Portability: compilers, datatypes

• Tools: debuggers, monitoring, performance

• And now: High Speed Networks and Grids

3

Computational Science Needs

• Complex real world problems require anincredible mix of technologies &expertise!

• Many different scientific & engineeringcomponents

• Many numerical algorithm components– Finite difference, Finite volume, Finite

elements, Spectral, …?

– Elliptic equations: multigrid, Krylovsubspace,…?

– Mesh refinement?

• Require large geo-distributed cross-disciplinary collaborations

Cactus User CommunityUsing and Developing Scientific Thorns

Goddard

Penn State

Wash UAEI

TAC

Tuebingen

Southampton

SISSA

Thessaloniki

Climate Modeling(NASA, Netherlands)

Chemical Engineering (U.Kansas)

Bio-Informatics(LSU, Bio-Grid, Canada)

Geophysics(Stanford)

Astrophysics(Zeus)

Crack Prop.(Cornell)

EU AstrophysicsNetwork

NASA Neutron StarGrand Challenge

Early Universe(LBL)

Portsmouth

RIKEN

AstrophysicalHydrodynamics

(LSU)

LSU

Austin

CFD (LSU)

Brownsville

Damage Mechanics(LSU)

Quantum Gravity(Hamilton)

JPL

German SFBConsortium

4

Cactus Developer CommunityDeveloping Computational Infrastructure

Argonne National

Laboratory

EGrid

Washington University

LSU CactusTeam

LawrenceBerkeley

Laboratory

Konrad-ZuseZentrum

NCSA

Clemson

SGI

Compaq

IntelMicrosoft

University Of Chicago

University Of Kansas

Cactus User Community

External Funding:

NSF, Max PlanckGesellschaft, DeutschesForschungsnetz, DOE,

NASA, NCSA, EuropeanCommission

TAC

Sun

Albert EinsteinInstitute

Hamilton College

GridLab

Tuebingen

Grand Vision

Remotemonitoring

from airport

Origin: NCSA

Remote Viz inSt Louis

T3E: Garching

Grid enabled Cactusruns on distributed

machines

Remote Viz andsteering fromBaton Rouge

Viz of data inSF café

Simulationscomposed &

launched fromCactus Portal

5

Remote Monitoring & Steering

• HTTPD module: anysimulation may act as itsown web server

• Communicate withsimulation using anybrowser

• Monitor, steer, visualize,collaborate

• Demo athttp://cactus.cct.lsu.edu:5555

Cactus Portal

• Collaboration focalpoint for a virtualorganization

• Interact, share data

• Start jobs onremote resources

• Move/browse files

• Track and monitorannounced jobs

• Access to new Gridtechnologies

• www.gridsphere.org

6

Notification

CactusPortal

SMSServer

MailServer

“The Grid”

Cactus IO and Viz

• Many Cactus appshave large scale dataneeds: O(TeraBytes)

• Currently support: 1-D,2-D, 3-D: to screen,ASCII, FlexIO, HDF5,Streaming HDF5,Panda parallel IO,Jpegs, Isosurfaces,Geodesics

• Remote viz, streamingdata

• Downsampling andZooming

• Checkpointing andRecovery

7

Remote Visualization & Steering

Remote Vizdata

HTT

P

Streaming HDF5Autodownsample

Remote Vizdata

• Cactus simulation starts, launched from portal

• Migrates itself to another site

• Registers new location

• User tracks/steers, using

http, streaming data, etc…

• Continues around Europe…

Cactus Worm: SC2000

8

Dynamic AdaptiveDistributed Computation

SDSC IBM SP1024 procs5x12x17 =1020

NCSA Origin Array256+128+1285x12x(4+2+2) =480

OC-12 line

(But only 2.5MB/sec)

GigE:100MB/sec17

12

5

4 2

12

5

2

Cactus + MPICH-G2Communications dynamically adapt

to application and environmentAny Cactus applicationScaling: 15% -> 85%

“Gordon Bell Prize”

(With U. Chicago/Northern,Supercomputing 2001, Denver)

Main Cactus BHSimulationstarts here

User only has to invoke Cactus“Spawner” thorn…

SC2001: Spawningon ARG Testbed

Appropriate analysis tasksspawned automatically to freeresources worldwide

9

Main Cactus BHSimulation startedin California

Dozens of low resolutionjobs sent out to testparameters

Data returned steers main jobHuge job generates remote data tobe visualized in Baltimore

SC2002:Task Farming

LRZ

Future Dynamic Grid Computing:New Scenarios

10

GridLab

• EU Project, $7M, 14 sites

• Motivated by previous Gridexperiments with Cactus

• Developing a GridApplication Toolkit (GAT)– co-development of

infrastructure with realapplications & users

– dynamic use of grids, self-aware simulations adapting totheir changing environment.

• Cactus is driving application!

Finally:

Cactus:Tom Goodale (CCT), Yaakoub Y ElKhamra (CCT), Dylan Stark (CCT), ThomasRadke (AEI), Jonathan Thornburg (AEI),John Shalf (LBL), Paul Walker, JoanMasso, Thomas Dramlitsch, GerdLanfermann + others

Links:

• Cactus: http://www.cactuscode.org

• GriKSL: http://www.griksl.org

• GridSphere: http://www.gridsphere.org

• GridLab: http://www.gridlab.org