P.O. Box 808 • Livermore • CA • 94551-0808Prepared by LLNL under Contract DE-AC52-07NA27344 LLNL-PRES-747356

PSAAP III Research Topic:Machine Learning and Scientific Applications

Katie Lewis (LLNL) and Mark Schraad (LANL)

Presented at the PSAAP III Bidders MeetingMinneapolis, MN on March 14, 2018

2LLNL-PRES-747356

Data science is one of the fastest growing areas of computing

Data science educationCyber and data science student programs

• Deep learning at scale• Multimodal semantic spaces• Science‐driven models 

Machine learning Graph and network analytics• Scalable 

community analysis

• Dynamic graphs• Random‐walk 

analysis• HPC 

implementation

Data-intensive HPCScaling to large, highly connected data

• Extended memory hierarchies• New computing technologies

3LLNL-PRES-747356

Global mapping of nuclear 

proliferation activities

Cognitive simulation systems

Predictive biology for 

human health and biosecurity 

Data scientists at work on challenging problems of national importance

4LLNL-PRES-747356

Incorporating Machine Learning into large‐scale simulations and experiments

NIF X-ray image

HYDRA simulation

Traditional pillar high-performance computing

Traditional pillarLarge-scale experiments

5LLNL-PRES-747356

Incorporating Machine Learning into large‐scale simulations and experiments

Traditional pillar high-performance computing

Traditional pillarLarge-scale experiments

New pillarMachine learning to compare

simulation and experiment

HYDRA simulation NIF X-ray imageComplete simulation and experiment data

Improved prediction

Deep neural network

6LLNL-PRES-747356

Key Objectives for Use of Machine Learning

Resource multiplier – Increase efficiency of resources, such as workforce, simulations, and compute performance, storage, etc.

Improve matching to experimental data – Combine learnedterms with known terms in physics models to better account for experimental discrepancies

Improve integration of multi‐scale/multi‐dimensional models –Model high‐fidelity coupling without greatly increasing computational costs

Need to evaluate varying applications:• Performance requirements: per time step vs. per simulation• Accuracy requirements: affect performance vs. affect results

7LLNL-PRES-747356

Example Applications

8LLNL-PRES-747356

Simulation surrogates to enable greater design exploration

Robust aspherical

design

Density at Bang TimeICF Ensembles team, LLNL

Machine learning helped find an asymmetric design predicted

to resist tent perturbations

60K simulations from Trinity Open Science were used to train network— Simulation surrogate allowed fast 

evaluation of combinations of parameters

— Parameters substantiated by detailed simulations not in the original dataset

Similar applications— Initial, low‐cost filter for simulations— Low‐cost evaluation of expensive physics 

models— Coupling multi‐scale simulations

Increasing Efficiency of Simulations and Workforce

Kelli Humbird, Luc Peterson, and Brian Spears

9LLNL-PRES-747356

Machine Learning predictions to steer ALE

Use Machine Learning techniques to predict and mitigate against tangling• Use physical quantities and mesh quality metrics

• Recognize local and global effects• Maintain accuracy and numerical stability

Similar applications• Adaptive Mesh Refinement• Automatic identification of interesting features to alert the designer

B" C"

A"D"

A"C"

B" D"

High Vor city

Increased Efficiency for Workforce

Brian Gallagher, Ming Jiang, Josh Kallman, et al

10LLNL-PRES-747356

Guide use of accelerators to reduce overall runtime 

rt3d_218M rt3d_652M rt3d_869M rt3d_1.1G rt3d_1.3G rt3d_2.6GCPU 0.555992 1.572962 2.231818 2.629638 3.156744 6.669277GPU 0.533135 0.898576 1.149787 1.442296 1.649144 3.658756SCoRE4HPC 0.208109 0.679062 0.810971 1.125081 1.378017 2.428869Thrust 0.357397 0.603257 0.804914 1.003872 1.20374 2.408579RAJA 0.47445 0.948078 1.245624 1.558656 1.870131 3.736115

0

1

2

3

4

5

6

7

8

Runtime (seconds)

Increase Efficiency of Compute Resources

Use reinforcement learning to maximize the use of accelerators while minimizing data movement

Potentially more adaptable to new hardware 

Similar applications• Dynamic load balancing• Optimize storage

Kevin Griffin

SCoRE4HPC learns to mimic Thrust solution without specific hardware knowledge

11LLNL-PRES-747356

Hybrid of physics and machine learning— Investigate backpropagation for corrections to 

coefficients— Corrections for closure model in Reynolds 

Averaged Navier‐Stokes (RANS) equations• Better capture transitional turbulence behavior

— Do not sacrifice physics knowledge

Use Machine Learning methodologies with physics models

Improving Matching to Experimental Data

12LLNL-PRES-747356

Multi‐scale and multi‐dimensional integration remains important and challenging

Surrogate models can be trained on higher‐fidelity physics models— Current tabular data can be replaced by models that more closely represent 

physical models— Complex workflows for integration with computationally expensive 

calculations may be reduced to function calls 

Surrogate models to increase simulation coupling

Ab-initio Atoms Long-time Microstructure Dislocation Crystal ContinuumInter-atomic forces, EOS,

excited states

Defects and interfaces, nucleation

Defects and defect

structures

Meso-scale multi-phase, multi-grain

evolution

Meso-scale strength

Meso-scale material

response

Macro-scale material

response

Improving Integration of Multi‐Scale, Multi‐Dimension

13LLNL-PRES-747356

Validation

14LLNL-PRES-747356

Dramatic success in machine learning has led to a torrent of Artificial Intelligence (AI) applications… However, the effectiveness of these systems is limited by the machine’s current inability to explain their decisions and actions to human users. – D. Gunning

https://www.darpa.mil/program/explainable‐artificial‐intelligence

Greater Confidence in Machine Learning is Necessary – “Explainable AI”

15LLNL-PRES-747356

Rigorous mathematical model accounting for:— Model fit to training data— Clustering of training data— Local slope of surface

Recognition of what training data is needed to improve model

Necessary to extend usage beyond “first pass” applications

How can we quantify and mitigate introduced uncertainty?

http://www.teraplot.com/3d-scatter-plot

16LLNL-PRES-747356

Can we use Machine Learning predictions to better understand physics?

“Automatically Learned Features”

x1 x2 x3 x4

h1 h2 h3

W(1)

g1 g2

W(2)

f1 f2

W(3)

Example: train deep learning network on images of faces

Input pixels

Edges at variousorientations

Object parts (combination of edges)

Object models

Traditional neural networks have limitations

What can the top layers of deep neural networks tell us about correlations?

How can these correlations be mapped to first principles?

[Example from Honglak Lee]

17LLNL-PRES-747356

How do we balance simulation data and experimental data?

Models trained on simulation data inherit the approximations of the simulations

Images may look different while numerical comparisons show similarities

Creating a better understanding of how much data is necessary and usefulness of transfer learning is essential

NIF Experiment1 HYDRA Simulation

*images are 200x200μm and have different color scales

17% Contour

18LLNL-PRES-747356

Research Opportunities

19LLNL-PRES-747356

Cross‐Cutting Research Opportunities

Physics—Potential new modeling techniques for increased fidelity at reasonable computation costs

Machine Learning—Different performance and scale requirements—Explainable AI

Statistics—Requires innovative, statistically‐driven models

20LLNL-PRES-747356

Conclusion

Several Machine Learning applications in ASC—Increase efficiency—Improving matching with experimental data—Improving multi‐scale and multi‐dimensional models

Results need to be explainable and add to our understanding of modeling

Provides for exciting, cross‐cutting research opportunities