Upload
matilda-richards
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
1
Applications
SURAgrid “All Hands” Meeting, Washington DCMarch 14 – 16, 2007
BioSim
Mahantesh Halappanavar,Ashutosh Mishra, Ravindra Joshi,
Mike Sachon
2
BioSim: Bio-electric Simulator for Whole Body Tissues
Numerical simulations for electrostimulation of tissues and whole-body biomodels
Predicts spatial and time dependent currents and voltages in part or whole-body biomodels
Numerous diagnostic and therapeutic applications, e.g., neurogenesis, cancer treatment, etc.
Fast parallelized computational approach
3
Simulation Models Whole-body discretized within a cubic space
simulation volume From electrical standpoint, tissues are characterized
as conductivities and permittivities Cartesian grid of points along the three axes. Thus, at
most a total of six nearest neighbors
* Dimensions in millimeters
4
Numerical Models
Kirchhoff’s node analysis
Recast to compute matrix only once
For large models, matrix inversion is intractable
LU decomposition of the matrix
0)]/}({/}{)/[( LAVdtVdLA
)]([]||][[ tBVVM tdtt
5
Numerical Models
Voltage: User-specified time-dependent waveform
Impose boundary conditions locally
Actual data for conductivity and permittivity
Results in extremely sparse (asymmetric) matrix
Red: Total elements in the matrix
Blue: Nonzero Values
[M]
6
Why Focus on Solvers? Scaling: (Source: David Keys, NIA Nov 2006)
– “Science” phase scales as: – “Solver” phase scales as – Computation will be almost all solver after several doublings– Optimal solver saves computation cycles for physics
)(NO
)( 23
NO
)(NO
7
The Landscape of Sparse Ax=b Solvers
Pivoting
LU
GMRES, QMR, …
Cholesky
Conjugate gradient
DirectA = LU
Iterativey’ = Ay
Non-symmetric
Symmetricpositivedefinite
More Robust Less Storage
More Robust
More General
Source: John Gilbert, Sparse Matrix Days in MIT 18.337
10
Computational Complexity
100 X 100 X 10 nodes: ~75 GB of memory (8-B floating precision)
Sparse data structure: ~ 6 MB (in our case) Sparse direct solver: SuperLU-DIST
– Xiaoye S. Li and James W. Dimmel, “SuperLU-DIST: A Scalable Distributed-Memory Sparse Direct Solver for Unsymmetric Linear Systems”, ACM Trans. Mathematical Software, June 2003, Volume 29, Number 2, Pages 110-140.
Fill reducing orderings with Metis– G. Karypis and V. Kumar, “A fast and high quality multilevel
scheme for partitioning irregular graphs”, SIAM Journal on Scientific Computing, 1999, Volume 20, Number 1.
11
Performance on compute clusters
144,000-node Rat Model
Blue: Average iteration time
Cyan: Factorization time
13
Output: Visualization with MATLAB
Simulated Potential Evolution Along the Entire 51-mm Width of the Rat Model
14
Deployment on
Mileva: 4-node cluster dedicated for SURAgrid purposes
Authentication – ODU Root CA– Cross certification with SURA Bridge – Compatibility of accounts for ODU users
Authorization Initial Goals:
– Develop larger whole-body models with greater resolution – Scalability tests
15
Grid Workflow
Establish user accounts for ODU users – SURAgrid Central User Authentication and
Authorization System– Off-line/Customized (e.g., USC, LSU)
Manually launch jobs based on remote resource – SSH/GSISSH/SURAgrid Portal– PBS/LSF/SGE
Transfer files – SCP/GSISCP/SURAgrid Portal
16
Recent Efforts in grid-enabling:
Porting to 100% open source tools (GCC/GFORTRAN)
SURAgrid Sites:– Texas A&M University: Calclab– University of Virginia: Grid04 and Grid11
Experiments with MUMPS 4– Symmetric matrices and out-of-core
Acknowledgements:– Jim Jokl, Steve Losen, Steve Johnson, Brain Brooks,
Kate Barzee and Mary Fran Yafchak
19
Conclusions
Science:– Electrostimulation has variety of diagnostic and
therapeutic applications– While numerical simulations provide many advantages
over real experiments, they can be very arduous
Grid enabling:– New possibilities with grid computing– Grid-enabling an application is complex and time
consuming– Security is nontrivial
20
Future Steps
Grid-enable BioSim– Explore alternatives for grid enabling BioSim– Explore funding opportunities– Load Balancing– Establish new collaborations – Scalability experiments with large compute clusters
accessible via SURAgrid Future applications:
– Molecular and Cellular Dynamics– Computational Nano-Electronics– Tools: Gromacs, DL-POLY, NAMD
21
References and Contacts
A Mishra, R Joshi, K Schoenbach and C Clark, “A Fast Parallelized Computational Approach Based on Sparse LU Factorization for Predictions of Spatial and Time-Dependent Currents and Voltages in Full-Body Biomodels”, IEEE Trans. Plasma Science, August 2006, Volume 34, Number 4.
http://www.lions.odu.edu/~rjoshi/ Ravindra Joshi, Ashutosh Mishra, Mike Sachon,
Mahantesh Halappanavar– (rjoshi, amishra, msachon, mhalappa)@odu.edu
22
Teaching Initiative
CS775/875: Distributed Computing
Ravi Mukkamala
Professor, Department of Computer Science
23
Details:
Graduate course with ~15 students Guest lecture Followed by a homework
– Familiarize with grid computing concepts– Hands-on approach– Experiment with Globus services & commands
Acknowledgements:– Jim Jokl, Steve Losen, Steve Johnson, Brain
Brooks, Nicole Geiger, Kate Barzee and Mary Fran Yafchak
25
Conclusions:
Laboratory for testing the concepts Potential to attract students For SURAgrid
– Large number of short-lived certificates– Cleanup … (CRLs?/home drives/…)– Centralized account creation (Still painful )– Short term funding/internships for grad/under-grad
students?