View
219
Download
2
Tags:
Embed Size (px)
Citation preview
Support for Adaptive Computations Applied to
Simulation of Fluids in Biological Systems
Kathy YelickU.C. Berkeley
Project Summary
• Provide easy-to-use, high performance tool for simulation of fluid flow in biological systems.– Using the Immersed Boundary Method
• Enable simulations on large-scale parallel machines. – Distributed memory machine including SMP
clusters
• Using Titanium, ADR, and KeLP with AMR• Specific demonstration problem: Simulation
of the heart model on Blue Horizon.
Outline
• Generic Immersed Boundary Method in Titanium (IBT)– Titanium status – Immersed Boundary method status
• Applications of the method– Heart model– Cochlea model
• Future Plans– Data analysis– Solvers
Titanium on Blue Horizon• Recent improvements:
– Support for PAPI (performance analysis)– Cache optimizations– Portable runtime layer (maintainable)– Faster LAPI-based implementation– Adapted to OS upgrade, ongoing issues
• Gcc does not on recent AIX’s• IBM C++ does not fully support templates
• Plans– Communication optimizations– Common runtime with UPC (possibly
CAF)
MPI vs. LAPI on the IBM SP
0
50
100
150
200
250
300
350
400
1 2 4 8
16
32
64
12
8
25
6
51
2
10
24
20
48
40
96
81
92
16
38
4
32
76
8
65
53
6
13
10
72
MB
/s IBM/MPI
IBM/LAPI
• LAPI bandwidth higher than MPI• Also better small-message overhead
– 9usec vs. 11usec
• Latest Titanium release leverages this
Immersed Boundary Method Structure
Fiber activation & force calculation
InterpolateVelocity
Navier-Stokes Solver
SpreadForce
4 steps in each timestep
Fiber Points
Interaction
Fluid Lattice
Immersed Boundary Method
• Recent Performance Improvements– Use of FFTW in Spectral solver
• 10x performance improvement on t3e• Use on BH still pending
– Use of scatter/gather communication• Copying bounding boxes is still faster• Depends on application and machine
– Load balancing• Alignment of fluid grid (in slabs) and fiber• Multigrid solver might offer more
possibilities
Load Balancing
Egg slicer Pizza cutter
0
50
100
150
200
250
300
350
400
450
0 5 10 15 20No. of processors.
MF
LO
PS
SpreadForce(Pizzacutter)
InterpolateVelocity(Pizzacutter)SpreadForce(Eggslicer)
InterpolateVelocity(Eggslicer)
Fluid grid is divided in slabs for 3D FFT
Application: Heart Simulation
05000
1000015000200002500030000350004000045000
Fiber Spreadforce
NS Solver Interpolatevelocity
Before
After
• Performance improvements over the last year
-64 node t3e ~= 2 node C90 ~= 1-node (8p) BH (probably)
Heart simulation on a Cray T3E
Heart Simulation
• Recent improvements– Support for heart input– Generate data for NYU visualization– Visualization is now OpenGL (ongoing)– Runs on
• Blue Horizon, NERSC SP, T3E, SGI, Millennium
• Short term plans– Finish checkpoint/restart– Larger runs– Complete performance model– Validation
Cochlea Model
• Model of the inner ear– Developed by Julian Bunn and Ed Givelberg
• Contains new features, e.g. membranes
• Implemented one of these last fall• Plan to have Givelberg here next year
Alpha Project Plans
• Support for new applications– Berkeley and Michigan
• Scaling– Berkeley
• Solvers– San Diego, Berkeley, LBNL
• Data analysis and model construction– Ohio, Maryland
Scallop: Multigrid Poisson Solver
• A latency tolerant elliptical solver library– Will be used to build Navier-Stokes Solver– Implemented in KeLP, with a simple interface
• Work by Scott Baden and Greg Balls– Based on Balls/Colella algorithm– 2D implementation in both KeLP and Titanium
• 3D Solver – Algorithm is complete– Implementation running, but performance
tuning is ongoing– Interface between Titanium and KeLP
developed
Elliptical solvers
• A finite-difference based solvers– Good for regular, block-structured domains
• Method of Local Corrections– Local solutions corrected by a coarse solution– Good accuracy, well-conditioned solutions
• Limited communication– Once to generate coarse grid values– Once to correct local solutions– Trades off extra computation for fewer
messages
KeLP implementation
• Advantages – abstractions available in C++– built in domain calculus– communication management– numerical kernels written in Fortran
• Simple interface– callable from other languages– no KeLP required in user code
Improved Heart Structure Model
• Current model is– Based on dog heart, textbook anatomy– Approximation by composing cones
• Building a more accurate model– Use modern imaging on human heart for model– Need to see individual fibers– Collaboration between
• Joel Saltz’s group and • Dr. Robert DePhilip in Anatomy Division of Biomedical
Informatics Dept. at Ohio State University
• Long term goal– Specialize model to patient using MRI data
Analysis of Cardiac Simulations
• Methods and tools to analyze 3D datasets from cardiac blood flow. – Outputs are velocity and pressure values on a
3D grid (1283) over many time steps.
• Characterize behavior under different pathological and physiological conditions. – Natural and artificial heart valves– Vary the initial values of
• Kinematic viscosity • Fluid density• Fiber stiffness and resting lengths
• Use parameter study to find conditions that may lead to aneurysm.
Analysis of Cardiac Simulations
• These queries require support for – spatial subsetting of one or more datasets – processing of the data of interest to
visualize and compare results from one or more datasets
– execution on high-performance machines and in a distributed environment.
• Implemented using DataCutter developed by Joel Saltz’s group.
Summary of Alpha Project Impact
• Several categories– Application development
• Heart and cochlea-component simulation
– Application-level package• Generic immersed boundary method• Parallel for shared and distributed memory• Enables new larger-scale simulations; finer grid
– Solver libraries• Method of Local Corrections• Improved scalability and load balance expected
– Data analysis• Building input data and analysis of results
applicationdata
softwaresystems