A massively parallel solid mechanics solver in ALYA

A massively parallel solid mechanics solver in ALYA

E. Casoni, G. Houzeaux, M. Vázquez and X. Sáez

JOCS - Càlcul d’altes prestacions en engineria

Barcelona, November 21th, 2013

CASE Department

CASE Department - Objectives

Computer Applications in Scienceand Engineering (CASE)Department

Interdisciplinary research unit of theBSC-CNS

Our mission:

To develop computational tools to simulate highly complex problemsseamlessly adapted to run onto high-end parallel supercomputers

Alya system - Large Scale Computational Mechanics

Alya is one of the two CFD codes of the PRACE benchmark suite.

Supercomputer Physical model CPUʼs

Lindgren (Sweden) NSI 12288

Huygens (The Netherlands) NSI 2128

Jugene BG (Germany) NSI, FSI 16384

Fermi BG (Italy) NSI + particles + species transp 16384

Curie Bullx (France) NSI, thermal flow 22528

Marenostrum (Spain) NSI , thermal flow, compressible 5000

Alya in PRACE

Impact for company Impact for BSC

- Simulation of large and complex tests: experimental tests unavailable!

- Demonstration of value of HPC to real world problems.

- Reduction of development cycle and cost. - Sensitivity analysis: more accurate results.

- Support for future product design and development. - Evolutions of simulation depending on the

industry requirements: need feedback to improve!- Analyze hypothetical scenarios.

HPC in Industry

Alya-SOLIDZ module

- Alya was initially designed for CFD: large meshes and expensive computations.

- The objective in Alya design is twofold:- To run with the highest efficiency standards in large scale supercomputing facilities.- To solve multi-physics problems in a coupled way.

Why Alya-SOLIDZ?

ITER vacuum vessel, property of F4E

- New challenges in solid mechanics: large structures, small details thatrequire fine meshes, ...

- Well-known robust commercial codes.. but need to be adapted formassively parallel architectures.

- No OpenSource codes.

- Need to solve more involved problems: coupling FSI, electro-mechanical,termo-mechanical, ...

The Alya-SOLIDZ module

ITER vacuum vessel, property of F4E

Main Alya

Solidz

Turnon DoIter Output

Initialize problem Set time step

METIS

Parallel service

Solver: Au = b

ImplicitExplicit

Constitutive model

Temper Nastin

SOLIDZ module

- Total Lagrangian formulation with FEM numerical method.

- Large deformations.

- Dynamic and static solution.

- Implicit (BCGStab, GMRES, Deflated CG) and Explicit schemes.

- Built-in material models (linear isotropic, neo-Hookean, Holzapfel-Ogden, inclusion of fibers, ...)

- Anisotropic materials (including orthotropic models).

- Plug-in interface to ABAQUS user-built subroutines.

- Overset meshes.

- Multiplier in order to refine the mesh.

Main features

Alya-SOLIDZ module

Fracture simulation

- Extended finite Element method (XFEM).

- Modeling the propagation of discontinuities (cracks) such that the path is independent of the mesh.

- Easily adapted to the code: does not modify the FEM routines.

- Open door to contact problems, fatigue...

Simulation of fracture process

Simulation of fracture process

- Implementation of fracture models for large scale simulations in composite laminate, focusing on the scalability of algorithms.

- Fracture simulation at the intra-laminar (XFEM) and inter-laminar (CZM) level.

- Validate models implementation against experiments.

-XFEM and CZM have been integrated in Alya with the aim of performing simulations of fracture in fiber laminates with different sequence.

Work done in collaboration with IMDEA materials and Oxford University

Alya-SOLIDZ module

Hybrid parallelization

- Multicore Processor is a processing system composed of two or more independentcores.

- Multicores run at lower frequencies, are more power efficient and get betterperformances than single cores, but their programming is difficult.

Recently, the most important trend in computer architecture is adding multiple cores inside a socket.

Parallelization: hybrid OpenMP / MPI

Hybrid parallel programming: mixing some parallel programming paradigms in order to exploit different levels of parallelism.

- Why? To exploit the task-level and the thread-level parallelism of multicore architecture.

- How? Assign a MPI-task to each node and an OpenMP thread per core.

- What do we win? Reduce the communication cost inside the node.

TaskShared memory

Parallelization: hybrid OpenMP / MPI

Real Industry case

Visualization team of CASE: Fernando Cucchietti and Guillermo Marín

Shear stress of a milling cutter punch (10M elements and 3000 cpu’s)

- Initial mesh of 500000 elements (multiplier)

- 3.9 Milion elements solved on 1000 cpu’s

- Hybrid mesh

- Run on Marenostrum III (48896 Inter Sandy Bridge processors in 3056 nodes)

Real Industry case: fusion reactor with magnetic loads

ITER vacuum vessel, property of F4E

Alya-SOLIDZ module

Coupled problems

- Avoid MPI barriers in order to reduce total time. - Improve the baseline speedup of 50%.- Reduce the communication between zones.

FSI coupling

FSI coupling: parallel strategy

force

The coupling code

- Alya architecture is modular: kernel, modules, and services. - Different solvers can be used on each module.- Coupling between modules by imposing continuity of variables.

Zonal strategy

- Load balancing between processors. - All the processors are working. - Define local communicators between zones for assembly and a global one for convergence check.

FSI coupling: scalability of the modules

FSI coupling: example

Pressure pulse in a compliant vessel

FSI coupling: example

Pressure wave propagation with the fluid deformed configuration.

Velocity of the fluid

FSI coupling: parallel strategy

Electro-mechanical model of the heart using medical images and mathematical models for the electric and mechanical coupling.

Cardiac- fluid-electro-mechanical coupling

Cardiac- fluid-electro-mechanical coupling

Cardiac- fluid-electro-mechanical coupling

- Using HPC systems will make numerical simulation closer to commercial computing and real-life simulations.

- Alya-Solidz module to solve large problems for nonlinear finite elements for continua and structures.

- New improvements to fit industry requirements: simulation of fracture process, coupling multi-physics, ...

- Hybrid version will take profit of the massively multicore architectures.

Concluding remarks

Thank you for your attention!