Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
Build your own simulation: The Einstein Toolkit
Frank Loffler
Center for Computation and TechnologyLouisiana State University, Baton Rouge, LA
Jul 30th 2011
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Solving Einstein’s Equations
Einstein equations:Gµν = 8πTµν
12 fully 2nd order PDE evolution equations4 coupled constraint equations4 gauge conditionsGR hydrodynamics, MHD, radiation transport
Fully numerical 3D models needed
Automated code generation
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Computational Requirements
Unigrid scales to hundreds of thousands of cores
Productions runs use ≈ 10 levels of mesh refinement, nested grids ofsize ≈ 60x60x60
Current mesh refinement runs scale up to ≈ 10k cores
Runtime from weeks to few months
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Key Challenge
Many scientific/engineering components
PhysicsMathematicsCFD
Many numerical algorithm components
Finite difference, finite volume, spectral methodsStructured or unstructured meshes, mesh refinementsMultipatch and multimodel
Many different computational components
Parallelism (MPI, OpenMP, ...)Parallel I/O (e.g. Checkpointing)Visualization
Challenge
Defining good abstractions to bring these together in a unified, scalableframework, enabling science
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
The Einstein Toolkit
Collection of software components and tools
Targeted at simulating and analyzing general relativistic astrophysicalsystems
Freely available as open source
See http://einsteintoolkit.org
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Guiding Principles
Open, community-driven software development
Separation of physics software from computational scienceinfrastructure
Well thought out and stable interfaces
Providing core computational tools to
facilitate interdisciplinary researchenable new sciencebroaden communitytake advantage of emerging Petascale computers and advancedcyberinfrastructure
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Current State
Last release on April 21, 2011
New releases roughly every six months
Atm. mostly Cactus (framework), Carpet (AMR)
Production quality features (Ninja, NRAR)
Not all new - grown from other projects
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Science Capabilities
McLachlan (BSSN, up to 8th order)
GRHydro (formerly based on WhiskyCode; Valencia formulation)
BH/NS initial data (TwoPunctures, Lorene)
Turduckening
MoL + e.g., Runge-Kutta
Carpet (Adaptive Mesh Refinement)
Black hole horizon finder (AHFinderDirect)
Gravitational wave extraction
Parallelization: MPI, OpenMP
Tools: e.g., EOS reader, HDF5 output, Visualization import
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Quality Control
Open source 6= offering download
Requirements for (new) Einstein Toolkit components:
sufficient quality (actually being used)documentation (e.g. publication)community interest (maintained)
Regular, tested releases
Providing easy step-by-step instructions for first-time users
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Einstein Toolkit testsuite status
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
People
Supported by NFS grants 0903973/0903782/0904015 (CIGR)
State-of-the-art set of tools for numerical relativity, open source
49 contributors over the past decade, both physics and CS
Currently 64 members from 24 sites in 10 countries
9 maintainers from 6 sites
> 200 publications, > 30 theses building on these components
Regular, tested releases
User support through various channels
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Tutorial for New Users
http://docs.einsteintoolkit.org/et-docs/Tutorial for New Users
Get account on cluster “QueenBee” (fill in web form)
Download (4 shell commands)
Configure (3 commands) [user name, email address, allocation]
Build (1 command)
Run simulation (1 command)
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Linda Holyoke - REU undergraduate student
Linda Holyoke: AMR visualization
Science vs. Simulation
Doing science >> Running a simulation
Need to know about meaningful initial conditions, numerical stability,accuracy/ resolution, have patience, have curiosity, have a “gutfeeling” for what is “right”...
Einstein Toolkit cannot give that!
But, with open codes that are easy to use, can concentrate on science!
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Convenience Tools
GetComponents Simfactory Formaline
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Tools: GetComponents
Task: Collect software from various repositoriesat different sites
Example simulation assembly:
Core Einstein Toolkit (svn.einsteintoolkit.org)
Cactus Flesh and Toolkit (svn.cactuscode.org)
Carpet AMR (carpetcode.org, hg)
Tools, Parameter Files and Data (svn.einsteintoolkit.org)
Group Modules (x.groupthorns.org)
Individual Modules (x.mythorns.org)
x: cvs, svn, darcs, git, hg, http
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Tools: Simulation Factory
http://www.simfactory.org/
Task: Provide support for common, repetitive steps:
Access remote systems, synchronize source code trees
Configure and build on different systems semi-automatically
Provide maintained list of supercomputer configurations
Manage simulations (follow “best practices”, avoid human errors)
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Tools: Formaline
Task: Ensure that simulations are and remain repeatable, rememberexactly how they were performed
Take snapshots of source code, system configuration; store it inexecutable and/or git repository
Tag all output files
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
General Guideline
The Einstein Toolkit
Does not want to provide “the” best codeRather wants to offer different codes with are
of high qualitymaintainedopen sourceeasy to use
→ Computational General Relativity for the Masses
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Support
Web site http://einsteintoolkit.org
Mailing list [email protected]
Bug tracking system http://trac.einsteintoolkit.org
Weekly public phone meetings
Also blog, wiki, code repositories, ...
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Future
Looking for users and contributions
Don’t want to take over software
Currently Cactus-centric, want to expand in other directions
Also looking for analysis tools / scripts(e.g. gravitational wave postprocessing tools)
Working on GRMHD
Improve existing numerical methods (scaling)
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Tutorial for New Users
Tutorial for New Users
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Prepare Tools
https://docs.einsteintoolkit.org/et-docs/Tutorial_for_New_Users
Setup on individual machines can be tricky
Usage of particular machine at LSU for this tutorial (QueenBee)
Need account? → http://einsteintoolkit.org/documentation/request
Using account: login (ssh)
Need git: soft add +git
Download GetComponents
wget --no-check-certificate \
https://github.com/gridaphobe/CRL/raw/ET_2011_05/GetComponents
chmod 755 GetComponents
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Download and Configure
Checkout (download) the Einstein Toolkit (may take a few minutes)
./GetComponents -a \
http://svn.einsteintoolkit.org/manifest/branches/ET_2011_05/\
einsteintoolkit.th
Includes Cactus, thorns, simfactory and example parameter files
Configure Simfactory
cd Cactus
cp simfactory/udb.example.pm simfactory/udb.pm
vi simfactory/udb.pm
Replace
YOUR_LOGIN\verb with your username on Queen [email protected] with your usual email addressYOUR_ALLOCATION with your project allocation on Queen Bee
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Build and Run
Compile Executable
./simfactory/sim build --thornlist=manifest/einsteintoolkit.th
Run a Simulation
./simfactory/sim create-submit static_tov \
--parfile=par/static_tov.par --procs=32 --walltime=8:0:0
Check simulation status
./simfactory/sim list-simulations
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Look at results
Make sure X-forwarding is enabled
ssh -Y qb.loni.org
Use gnuplot to visualize data
cd $SCRATCH/simulations/static_tov/output-0000/static_tov
gnuplot
p ’hydrobase::rho.maximum.asc’ u 2:3 w lp
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Additional Example Parameter Files
par/ks-mclachlan.par:single isolated rotating black hole
qc0-mclachlan.par:binary black hole system using the moving puncture technique(12 orbit
)
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30
Future
Looking for users and contributions
Don’t want to take over software
Currently Cactus-centric, want to expand in other directions
Also looking for analysis tools / scripts(e.g. gravitational wave postprocessing tools)
Working on GRMHD
Improve existing numerical methods (scaling)
Frank Loffler Build your own simulation: The Einstein Toolkit 2011-07-30