Software Integration Status and Plans Gbor Tth UM CRASH Team,
Ann Arbor, MI October 29, 2010
Slide 2
Page 2 This talk is a status update and part of our response to
the review team recommendations in the V&V area Outline of Talk
Software architecture and modeling schema New algorithms and work
in progress Testing strategy Example tests Version Control
Documentation UQ integration discussed by Paul Drake and James
Holloway
Slide 3
Page 3 CRASH Initialization Software Architecture and Current
Algorithms HYADES Multi-material hydro with EOS Multi-group flux
limited diffusion Electron heat conduction Laser heating 1D or 2D
Lagrangian More flexible grid options CRASH BATSRUS 5-material
hydro with EOS Flux-limited gray/multi-group diffusion Flux limited
electron heat conduction 2D AMR (RZ) or 3D AMR (XYZ) Split
semi-implicit time stepping Blurred synthetic radiograph Laser
package, external EOS tables, Improve parallel I/O. Adjoint method.
PDT Multi-group discrete ordinates 3D or 2D RZ adaptive grid
Implicit time stepping, TBDF2 Krylov methods for absorption Nearly
optimal sweep scheduling Multigroup diffusion, improved I/O
Parallel comm.: (rho,u,p,T e,m)(x,y,z) Parallel comm.: (E r,S re,S
rm )(x,y,z) Physics informed emulator Data reduction Flat file(s):
( ,u,p,T e,T r,E r,g,m)(x,r) Notation blue color: work in progress
red color: added this year Postprocessor statistical analysis
UQ
Slide 4
Page 4 New Algorithms Full energy conservation for
electron/radiation physics Total energy update but apply slope
limiter on primitive variables Electron flux limiter limit
Spitzer-Harm flux by fraction of free-streaming heat flux
5-material EOS and opacity calculations Xe, Be, Au, acrylic,
polyimide Use levelset with signed distance Block Adaptive Tree
Library (BATL) Efficient dynamic AMR in 1, 2 and 3D Split
semi-implicit scheme Solve each energy group and electrons
independently. Requires less memory. Allows PCG. Sometimes faster.
Synthetic radiographs with blurring Add Poisson noise due to finite
photon count. Smooth at the scale that corresponds to the pinhole
size.
Slide 5
Page 5 Work in Progress Laser package Alternative to HYADES.
Full control. Allows 3D initialization. External EOS/Opacity tables
Alternative to the inline EOS/opacity calculations Collaboration
with Marcel Klapisch and Michel Busquet Multi-level preconditioning
Using HYPRE library Should scale better than BILU preconditioner.
Adjoint method Reverse execution of CRASH simulation with a series
of restart files. Provides sensitivity information. Implemented but
needs to be tested. Improved parallel I/O options HDF5, striped I/O
access. Prototype implemented. Allows visualization with various
software packages. Potentially faster output from code. More robust
feature recognition in radiographs Based on thresholds and integral
quantities
Slide 6
Page 6 CRASH Adjoint Solver Purpose of the adjoint Accelerate
UQ through efficient calculation of the sensitivity of an output to
a large number of inputs and parameters
Slide 7
Page 7 CRASH Adjoint Solver Approach Discrete unsteady adjoint
Not a separate code branch New versions of routines that compute
linearized reverse steps Status Restart file checkpointing
implemented and tested Adjoint step implemented in BATSRUS for the
explicit solver Testing in progress Changes for semi-implicit CRASH
solver outlined Cost Additional storage and computation compared to
forward mode For 10 6 time steps and logarithmic checkpointing:
Storage of a maximum of 300 checkpoints at one time Recomputation
factor of 3
Slide 8
Page 8 Verification and Testing New program units are
implemented with unit tests Nightly execution of many unit tests
New features are implemented together with verification tests Daily
(18-hour) verification & full system tests are run on a 16-core
Mac. Tests should cover all aspects of the new feature including
restart. Using grid convergence studies and model-model comparison.
Compatibility and reproducibility of features are checked with the
functionality test suite Nightly runs ensure that bugs are
discovered early Running on 8 different platforms/compilers on 1 to
4 cores: tests portability Parallel Scaling Tests Weekly scaling
test on 128 and 256 cores of hera. Many-core runs require DAT.
Reveals software and hardware issues. Confirms that results are
independent of the number of cores.
Slide 9
Page 9 Nightly unit and functionality tests: 96 SWMF tests 65
BATSRUS tests 4 unique to CRASH including restart
Slide 10
Page 10 29 verification tests: pass if converge at the correct
rate 15 full system tests: pass if results have not changed
includes restart Daily CRASH Verification and Full System
Tests
Slide 11
Page 11 Verification Test: electron physics Initial condition
is a semi-analytic radiation solution of R. Lowrie Use electron EOS
that mimics the radiation energy: E e =aT e 4 Energy exchange and
heat conduction follow Kramers formula: T e -3.5 Solution is a
supercritical Mach 5 shock with a thermal front advected on a
rotated AMR grid
Slide 12
Page 12 Verification Test: 1D electron physics Thermal front
Hydro shock relaxation
Slide 13
Page 13 Full system test:5 materials and 30 groups with
electron physics on R-Z grid. 6 micron effective resolution with 1
AMR Unsplit vs. split semi-implicit schemes: 2017s vs. 1167s
wall-clock time Blurred radiograph
Slide 14
Page 14 Full System Test: 3D gray with elliptical nozzle Y = 0
cut Synthetic radiograph from Y Z Y Synthetic radiograph from Z Z =
0 cut
Slide 15
Page 15 Strong Parallel Scaling of CRASH 2.1 3D semi-implicit
multigroup-diffusion and heat conduction. 2 levels of dynamic AMR,
2.6 million cells in 4x4x4 blocks. Hera Pleiades
Slide 16
Page 16 Weak Parallel Scaling of PDT on hera Per core: 2048
cells with 8 spatial unknowns, 80 directions and 10 energy groups.
1 to 12288 cores (maximum of 1.6 10 11 unknowns)
Slide 17
Page 17 Weak Parallel Scaling of PDT on hera
Slide 18
Page 18 Version Release Development is done in a single CVS
branch (HEAD version) Code version number Incremented when code
behavior is significantly modified Saved with runs Code is tagged
in CVS before and after major changes Allows recovery of previous
version Allows comparison of results, performance, portability etc.
Release: stable versions are created as CVS tags or branches
Released versions 1.0, 1.1, 2.0 and 2.1 so far. Includes BATSRUS,
CRASHTEST and CRASH_data repositories. The released stable versions
are used by UQ.
Slide 19
Page 19 Several Layers of Documentation CRASH and UQ primers
Documentation of new algorithms Developed during preliminary
discussions added to CVS when code is committed Software
Development Standards document Quality requirements, testing
procedures, data naming etc. Documentation of changes in the CVS
logs Required for every change, generates an email Documentation in
the source code XML description of input parameter produces most of
the user manual User manual describes full usage including examples
CRASH full system tests serve as working examples
Slide 20
Page 20 Concluding Remarks We have made substantial progress
with our algorithms We use good software engineering practices Unit
tests are developed with new code Nightly functionality tests Daily
verification test suite covering all aspects of CRASH physics All
CVS changes generate notification email to developers Future
algorithmic development Driven by efficiency and functionality
requirements Driven by UQ Interaction with UQ software Through
scripts