Nov. 2002 NERSC/LBNL 1

Climate Modeling: Coupling Component Models by MPH for Distributed Multi-Component Environment

Chris Ding and Yun (Helen) He

NERSC Division

Lawrence Berkeley National Laboratory

Nov. 2002 NERSC/LBNL 2

Nov. 2002 NERSC/LBNL 3

Motivation

Application problems grow in scale & complexity

Effective organization of simulation software system a major issue

Software lasts much longer than a computer!

Nov. 2002 NERSC/LBNL 4

Multi-Component Approach Build from (semi-)independent programs Coupled Climate System = Atmosphere + Ocean + Sea-Ice

+ Land-Surface + Flux-Coupler Components developed by different groups at different

institutions Maximum flexibility and independence Algorithm, implementation depends on individual groups,

practicality, time-to-completion, etc. Components communicate through well-defined interface

data structure. Software industry trend (CORBA, DCE, DCOM) Common Component Architecture (DOE project) ESMF

Nov. 2002 NERSC/LBNL 5

Distributed Components on HPC Systems

Use MPI for high performance MPH: establish a multi-component environment

MPI Communicator for each components Component name registration Resource allocation for each component Support different job execution modes Stand-out / stand-in redirect Complete flexibility

Similar to PVM, but much smaller, simpler, scalable, and support more execution / integration modes.

Nov. 2002 NERSC/LBNL 6

A climate simulation system consists of many independently-developed componentson distributed memory multi-processor computer

Single-component executable: Each component is a stand-alone executable

Multi-component executable: Several components compiled into an executable

Multi-component system execution modes: Single-Component executable Multi-Executable system (SCME) Multi-Component executable Single-Executable system (MCSE) Multi-Component executable Multi-Executable system (MCME)

Nov. 2002 NERSC/LBNL 7

Component Integration / Job Execution Modes Single-Component exec. Multi-Executable system (SCME):

Each component is an independent executable image Components run on separate subsets of SMP nodes Max flexibility in language, data structures, etc. Industry standard approach

Multi-Component exec. Single-Executable system (MCSE): Each component is a module All components compiled into a single executable Many issues: name conflict, static allocations, etc. Stand-alone component Easy to understand and coordinate

Nov. 2002 NERSC/LBNL 8

Component Integration / Job Execution Modes

Multi-Component exec. Multi-executable system (MCME): Several components compiled into one executable Multiple executables form a single system Different executables run on different processors Different components within same executable could run on

separate subsets of processors Maximum flexibility Includes SCME and MCSE as special cases Easy to adopt for concurrent ensemble simulations

Nov. 2002 NERSC/LBNL 9

Multi-Component Single-Executable (MCSE)

master.Fmaster.F PCM PCM call call MPH_setup_MCSEMPH_setup_MCSE ( ( ‘ ‘atmosphereatmosphere’, ! ‘atmosphere’ registered’, ! ‘atmosphere’ registered ‘ ‘oceanocean’, ! ‘ocean’ registered’, ! ‘ocean’ registered ‘ ‘couplercoupler’, ! ‘coupler’ registered’, ! ‘coupler’ registered ) ! Add more components) ! Add more components

if (PE_in_component (‘if (PE_in_component (‘oceanocean’, ’, commcomm)) call )) call ocean_v1ocean_v1 ( (commcomm)) if (PE_in_component (‘if (PE_in_component (‘atmosphereatmosphere’, ’, commcomm)) call )) call atmosphereatmosphere ( (commcomm)) if (PE_in_component (‘if (PE_in_component (‘couplercoupler’, ’, commcomm)) call )) call coupler_v2coupler_v2 ( (commcomm))

PROCESSOR_MAPPROCESSOR_MAP atmosphereatmosphere 0 7 0 7 ocean ocean 8 13 8 13 couplercoupler 14 15 14 15

Nov. 2002 NERSC/LBNL 10

Single-Component Multi-Executable (SCME)

Coupled System = Atmosphere + Ocean + Flux-Coupler

atm.F: atm.F: call MPH_setup (“call MPH_setup (“atmosphereatmosphere”, ”, atmosphere_Worldatmosphere_World)) ocean.F: ocean.F: call MPH_setup (“call MPH_setup (“oceanocean”, ”, ocean_Worldocean_World)) coupler.F: coupler.F: call MPH_setup (“call MPH_setup (“couplercoupler”, ”, coupler_Worldcoupler_World))

Component Registration File:Component Registration File: atmosphereatmosphere ocean ocean couplercoupler

Nov. 2002 NERSC/LBNL 11

MPH3: Multi-Component Multi-Executable (MCME)

mpi_exe_world = mpi_exe_world = MPH_componentsMPH_components (name1=‘ (name1=‘oceanocean’, name2=‘’, name2=‘iceice’,…) ’,…) Component Registration File:Component Registration File: BEGINBEGIN couplercoupler Multi_Comp_StartMulti_Comp_Start 22 ocean 0 3ocean 0 3 ice 4 10ice 4 10 Multi_Comp_EndMulti_Comp_End Multi_Comp_StartMulti_Comp_Start 33 atmosphere 0 10 atmosphere 0 10 land 11 13land 11 13 chemistry 14 25chemistry 14 25 Multi_Comp_EndMulti_Comp_End ENDEND

Launch parallel job on IBM SP: @ task_geometry = {(5,2)(1,3)(4,6,0)}Launch parallel job on IBM SP: @ task_geometry = {(5,2)(1,3)(4,6,0)}

Nov. 2002 NERSC/LBNL 12

Joining two components

MPH_comm_join (“atmosphere”, “ocean”, comm_new) comm_new contains all nodes in “atmosphere”, “ocean”. “atmosphere” nodes rank 0~7, “ocean” nodes rank 8~11

MPH_comm_join (“ocean”, “atmosphere”, comm_new) “ocean” nodes rank 0~3, “atmosphere” nodes rank 4~11

Afterwards, data remapping with “comm_new”

Direct communication between two components MPI_send (…, MPH_global_id (“ocean”, 3), …)

for query / probe tasks

Nov. 2002 NERSC/LBNL 13

MPH Inquiry Functions

MPH_global_id() MPH_local_id() MPH_component_name() MPH_total_components()

Nov. 2002 NERSC/LBNL 14

Status and Users Completed MPH1, MPH2, MPH3

Software available free online: http://www.nersc.gov/research/SCG/acpi Complete user manual

MPH runs on IBM SP SGI Origin HP Compaq clusters PC Linux clusters

MPH users NCAR CCSM CSU Icosahedra Grid Coupled Climate Model

People expressed clear interests in using MPH SGI/NASA: Irene Carpenter / Jim Laft, on SGI for coupled models UK ECMWF, for ensemble simulations Johannes Diemer, Germany, for coupled model on HP clusters

Nov. 2002 NERSC/LBNL 15

Summary

Multi-Component Approach for large & complex application software

MPH glues together distributed components Name registration, resource allocation, standard-

out Single-Component Multi-Executable (SCME) Multi-Component Single-Executable (MCSE) Multi-Component Multi-Executable (MCME)