Welcome & Overview of the Workshop

Welcome &Welcome &Overview of the WorkshopOverview of the Workshop

Kengo Nakajima

International Workshop for Large-Scale Coupled Simulations: Computation and Physics April 9, 2007, Earthquake Research Institute, The University of Tokyo

Department of Earth & Planetary ScienceThe University of Tokyo and JST/CREST

CREST WS APR-09-2007

2

Acknowledgements

• Sponsor– CREST, Japan Science and Technology Agency (JST)

• Invited Speakers• Local Arrangements

– Prof. Takashi Furumura (ERI, The University of Tokyo)– Dr. Tatsuhiko Saito (ERI, The University of Tokyo)– Prof. Hiroshi Okuda (RACE, The University of Tokyo)


3

Overview of this talk

• Overview of the Project– Integrated Predictive Simulation System for Earthquake

and Tsunami Disaster• Brief overview of Coupling• Overview of this Workshop


4


5

Integrated Predictive Simulation System for Earthquake and Tsunami Disaster

• PI (Principal Investigator)– Prof. Mitsuhiro Matsu’ura (The University of Tokyo)

• 5-year National Project– Oct. 2005 - Mar. 2011– CREST, Japan Science and Technology Agency (JST)

• Basic Research Programs• Category: Integrated Simulations for Multiscale/Multiphysics


6

Institutions

• The University of Tokyo– Department of Earth & Planetary Science (EPS)– Earthquake Research Institute (ERI)– Research into Artifacts, Center for Engineering (RACE)

• Tokyo Institute of Technology (Titech)• Sophia University• National Research Institute for Earth Science and

Disaster Prevention (NIED)• Geographical Survey Institute, Japan (GSI)


7


8

Features of the Project (1/3)• The first integrated simulation system for prediction of

earthquake and tsunami disasters, which covers entire series of multi-scale processes, such as: – Plate deformation– Dynamic fault rupture– Seismic wave/tsunami propagation– Oscillation of buildings

• Target hardware– Earth Simulator (ES), PC clusters


9


10

Features of the Project (2/3)

• The simulation is complimented by large number of observational data sets obtained through nation-wide network of seismic instruments and GPS etc. – Crustal movement: GEONET– Seismic activity: Hi-net– Strong Motion: K-NET/KiK-net

• >1,000 points for each network – every 20 km


11


12


13

Features of the Project (3/3)• Both of advanced simulation models and

infrastructure for large-scale simulations are developed.– Platform– GRID-like Environment

• Strong Collaboration/Integration between …– Simulation and Observation– Geophysics and Computer/Computational Science– Earthquake Science and Earthquake Engineering– Simulation models– Observation/Data assimilation– Infrastructure (Platform+Database)


14


15


16


17


18


19

Challenges

• Integrated predictive simulation system for earthquake and tsunami disaster– Integration of simulation and observation

• Prof. Matsu’ura’s talk in this afternoon

– Coupled simulations• Talks by members in this afternoon

• Current Status– Each simulation model/code has been

developed/updated, optimized for the Earth Simulator, parallelized for large-scale simulation (separately).


20

• Overview of the Project–Integrated Predictive Simulation System

for Earthquake and Tsunami Disaster• Brief overview of Coupling• Overview of this Workshop


21

CouplingStrong/Tight Coupling

PDE’s to be solved simultaneouslySingle (special) codeRobust for strongly nonlinear problemse.g. Flow with Chemical Reaction

Weak/Loose CouplingPDE’s (PDE sets) can be solved separatelyIndividual framework is keptFlexibleApproach in this project

⎭⎬⎫

⎩⎨⎧−

=⎭⎬⎫

⎩⎨⎧⎥⎦

⎤⎢⎣

⎡+

⎭⎬⎫

⎩⎨⎧⎥⎦

⎤⎢⎣

⎡−

+⎭⎬⎫

⎩⎨⎧⎥⎦

⎤⎢⎣

⎡0M1

φU

K00K

φU

0SSC

φU

M00M αρ

LT

L &

&

&&

&&

)( ttttttttt

Δ+−−=++Δ+Δ+Δ+

αρ M1φSKUUCUM &&&&

tttttt TLL

Δ+Δ+Δ+

=+ USφKφM &&&


22

Weak/Loose Coupling

with Interactioniterations between 2 (or more ) processes for accurate/stable solutionGauss-Seidel, Jacobi，Newton-Raphson

PDE’s#1

PDE’s#2

PDE’s#1

PDE’s#2

without Interaction (One-Way)

No interactionSuitable if time/grid scale is very different each other.


23

“Centralized” or “Distributed”

Dr. Drummond’s Talk


24

Final Goal ofCoupling Library/Framework

to provide interpolation/extrapolation between two meshes


25

Requirements for Coupling Library/Framework

Accuracyvarious types of inter/extrapolation/averaging methods

EfficiencyPortabilityFlexibility

distributed data sets, pure distributed manner (NO centralization)various methods: FEM, FDM, BEM etc.

Easy-to-Usesubroutine, function call’ssmall changes to original codes

Independent from Physics’start from special cases …


26

• MpCCI– Mesh-based Parallel Code Coupling Interface– Fraunhofer/SCAI（former GMD/SCAI）– MPI-based, MPI-like interface for C & FORTRAN– Coupling of commercial codes

• CCA – Common Component Architecture– Environment for development of parallel applications

• Interoperability, Coupled simulations: “Wrapper”

– Dr. Allan’s talk in this afternoon• DCT

– Distributed Coupling Toolkit (DCT)– Dr. Drummond’s talk in this morning

Libraries for Coupled Simulations


27

• Geophysics: Atmospheric/Ocean Science– OASIS

• PRISM（Program for Integrated Earth System Modeling）：EU

– FDM, Spectral Methods• Common data structure

Libraries for Coupled Simulations (cont.)


28

• SIAM Conference on Parallel Processing for Scientific Computing 2006（PP06）

• SIAM Conference on Computational Science and Engineering 2007 (CSE07)

• More than 10 mini-symposiums (40 presentations) for coupling libraries/frameworks in each conference

• In SIAM/CSE07– Material Science

• bridging between macro- (FEM) and micro- (MD) simulations

• ICCM 2007 at Hiroshima (last week)

Activities in SIAM (Society for Applied Mathematics) Conferences


29

What I am understanding on current frameworks for coupled simulations

• Integrated frameworks for development of MPI-based parallel applications on MPP/PC clusters and/or grid environments.– “Coupling” is one capability provided by these.

• Interoperability is the most important issue.– easy interface for various applications C/FORTAN/JAVA etc.

• They seem very nice, but sometimes it seems difficult to apply the “coupling” capability to existing “parallel”applications (e.g. ours…).– We know that the initial idea/design of these frameworks

started from some specific applications.


30

Features of “OUR” Applications (1/2)

• Various types of methods– FEM, FDM, BIEM

• Most of the codes have been parallelized using MPI (some of them have been optimized on the Earth Simulator).

• Most of the codes are written in FORTRAN• Communications between applications occur at only

boundaries– small amount of communication meesages


31

Features of “OUR” Applications (2/2)

• Wide range of “time-scale’s” for applications.• Loosely coupling works well … Mostly “One-

Way” is enough.

GEONET Hi-net K-NET

DB for topographical information

(plates, ocean-floor, underground)

Crustal Movement

Crustal Movement Seismic Activity Strong Motion

Platform for Integrated SimulationData Processing, Visualization, Linear SolversPC clusters for small-intermediate problem sizeEarth Simulator for large-scale problems

StressAccumulation Dynamic Rupture

Wave Propagation

TsunamiSimulation Models

Observational DataProcessing & Assimilation

SimulationResults

Platform and DB’s

BuildingOscillation

GISUrban Information


32

“OUR” Strategy for Coupling• Our own framework for coupled simulations is

developed in this project.– to support communications of various types of

applications for various types of geometries on boundaries, such as vertices, elements, surfaces etc.

• Mainly focused on FORTRAN90– to make use of capability of “module” and “structure” of

FORTRAN90– basically called “from” FORTRAN90

• Applications written in “C” are called from “main program” written in FORTRAN (My talk in this afternoon)

– No “OO” features: ES• SPMD (Single Program Multiple Data) programming

paradigm with “M×N” data redistribution.


33

• Overview of the Project–Integrated Predictive Simulation System

for Earthquake and Tsunami Disaster• Brief overview of Coupling• Overview of this Workshop


34

First International Symposiumhttp://www-solid.eps.s.u-tokyo.ac.jp/~crest/sympo2006

• October 23 & 24, 2006• Koshiba Hall, University of Tokyo• Very interesting conference

– participants from both of geophysics and computer/computational science communities

• Presentation materials are available from web-site.


35

Speakers of this WS• Invited (international) speakers

– Computer Science• Numerical Algorithms, Parallel Coupling Libraries, PSE etc.• Participants of “ICCM 2007 (International Conference on

Computational Methods”, Apr.4-6, 2007, Hiroshima)– Organized Workshop: Innovative Computational Strategies for

Parallel/Grid Environments (H. Okuda, K. Nakajima)

– Many experiences in collaborations with application communities

• Invited (domestic) speaker (Dr.Sugimoto)– Large-scale simulation

• “Revolutionary Simulation Software (RSS21)”• another national project

• Other (domestic) speakers– Earth Science (some exceptions)– Members of the CREST project


36

Session 1: Opening and Invited Talks (I) 09:30-12:05 09:30-09:50 Kengo Nakajima (University of Tokyo, Japan) 09:50-10:35 Osni Marques (Lawrence Berkeley National Laboratory, USA) 10:35-11:20 Tony Drummond (Lawrence Berkeley National Laboratory, USA) 11:20-12:05 Lutz Gross (ESSCC, The University of Queensland , Australia)

(Lunch Break) 12:05-13:30

Session 2: Activities in CREST Project 13:30-15:50 13:30-14:05 Mitsuhiro Matsu'ura (University of Tokyo, Japan) 14:05-14:40 Chihiro Hashimoto (University of Tokyo, Japan) 14:40-15:15 Takashi Furumura (Earthquake Research Institute, University of Tokyo, Japan) 15:15-15:50 Kengo Nakajima (University of Tokyo, Japan)

(Coffee Break) 15:50-16:10

Session 3: Invited Talks (II) 16:10-17:30 16:10-16:45 Shin'ichiro Sugimoto (Kyushu University, Japan) 16:45-17:30 Benjamin A. Allan (Sandia National Laboratories, USA)

(Closing Remarks) 17:30-17:45

Program


37

Goal: Integration through actual works

• “Integration/coupling” is the key-word of this project– Multiphysics– Simulations & Observations– Physics & Computer/Computational Sciences– Earthquake Science & Earthquake “Engineering”

• This workshop is focusing on "coupling" of multi-physics and simulations/observation.

• We hope this is a good opportunity for information exchange and discussions for future directions of our project.

Documents

Welcome & Overview of the Workshop