High Performance Computer Simulations - · PDF fileHigh Performance Computer Simulations Borrajo Juan*, ... “Battle of Speed Machines”, ... CFD Calculation X8

High Performance Computer Simulations

Borrajo Juan*, Chimenti Martín*, De Vecchi Hugo⁺, Grioni Mauro#, Rojido Martín⁺, Seres Vincent^

* IMPSA HYDRO, ⁺ IMPSA CIT, # IMPSA WIND, ^ IMPSA IT

• The only company with proprietary technology for hydropower and wind power generation in Latin America.

• The largest wind generator manufacturer in Latin America and one of the biggest hydropower manufacturers in the region.

• The largest investor in wind farms in Latin America with assets in Brazil, Uruguay, Argentina and Peru.

• The world’s fastest growing family business according to the Top 50 Global Challengers perform by Ernst & Young.

• A company with a high profit track record for more than 106 years.

• A major source of qualified and sustainable work.

• Local and experienced management teams.

• Worldwide exporter in the Hydropower business and for all Latin America in wind power generation.

• A company with access to international capital markets for over 15 years and financed by multilateral credit banks (BID, CAF, CII).

IMPSA is…

2

IMPSA - Production centers

ARGENTINA MALAYSIA

IMPSA Plant I Mendoza - Argentina

IMPSA Hydro Recife Suape, Brazil

IMPSA Wind Recife Suape, Brazil

IMPSA Hydro Recife Suape, Brazil

BRAZIL

IMPSA Wind Recife Suape, Brazil

IMPSA Plant II Mendoza - Argentina

IMPSA Malaysia Malaysia

IMPSA Malaysia Malaysia

4

* Article published in the Financial Times, “Battle of Speed Machines”, on Wednesday, July 10, 2013.

Each 15 years, speed of the most powerful computers in the world are multiplied by 1000.

HISTORICAL DEVELOPMENT

6

• IMPSA hardware has followed a similar pattern.

• Historically, the number of degrees of freedom (DOF) was used as a reference.

• 15 years ago, on a HP 7000, for FEM analysis, it was possible to run 100,000 (one hundred thousand) DOF.

• Nowadays, we can analyze more than 100,000,000 (one hundred million) DOF.

• In 15 years, the size of the models created and resolved by IMPSA has been multiplied by 1000.

HISTORICAL DEVELOPMENT - IMPSA

7

IMPSA HARDWARE - Workstations

Personal Computer

X1 (26 Users)

Nodes 1 Dell 7500

Processors 2 Xeon® X5647 2.93 GHz

Cores 8 - 2 CPU, 4 cores / CPU

RAM 24 GB

ROM 500GB - 7.2Krpm, SATA 2

Network Ethernet 1 Gbps

FEM Calculation

X2 (1 Workstation)

Nodes 1 Dell 7500



RAM 48 GB

ROM 1TB - 7.2Krpm, SATA 2


CFD Calculation

X8 (1 Cluster)

Nodes 8 Dell 7500



RAM 18 GB/Node = 144 GB

ROM 4TB - 7.2Krpm, SATA 2


2007 2009/2012

8

IMPSA HARDWARE - Cluster HPC

Cluster Dell HPC Rack : PowerEdge 4220 HPC Nodes : 8 vertical blades, PowerEdge M620 Head Nodes : 2 horizontal blades Storage Bay : 30 HDD, SAS 2.0, 10krpm, de 900GB

= 27 TB

Nodes 8 PowerEdge M620

Processors 16 Xeon® E5-2680 2.70 GHz, 331 Gflops

Cores 128 - 16 CPU, 8 cores por CPU

RAM 32 GB/CPU = 512 GB

ROM 2 x 146 GB - 15Krpm SAS 2.0

Network Infiniband 40 Gbps

2013 (2.77 Tflops) 9

COMPUTER APPLICATIONS AND CHALLENGES

SPEED

ACCURACY

LARGE ASSEMBLIES

FLUID DYNAMICS TRANSIENTS

MULTIPHYSICS SIMULATIONS

10

MECHANISMS AND ASSEMBLIES WITH NONLINEAR CONTACT

Assembly: Head Cover, Wicket Gate, Bottom Ring, Spiral Case

Simulation with nonlinear contacts

Large displacements

2 Millions DOF

12 Iterations

SPEED

11

SPEED

ACCURACY

LARGE ASSEMBLIES




12

ASSESMENT OF DISPLACEMENTS AND STIFFNESS – HEAD COVER

ACCURACY

• Criticality in large models is given by the amount of available RAM.

• On certified Clusters, the use of Virtual Memory is not allowed, so, the size of the model to run is limited.

• On Workstations, because the use of Virtual Memory is allowed, it is possible to run larger models than those runnable on the Cluster.

13

Model with 114 Million degrees of freedom

ACCURACY

14

Elapsed time for resolution: 9 hours, 20 minutes (on a Workstation)

SPEED

ACCURACY

LARGE ASSEMBLIES




15

ASSEMBLY MODEL, EMBEDDED PARTS AND CONDRETE, PORCE III

LARGE ASSEMBLIES

Interactions between different components can lead to design solutions that would not be possible in an isolated analysis

16

INTERACTION ELECTROMECHANICAL EQUIPMENT - POWERHOUSE

LARGE ASSEMBLIES

Linear-elastic analysis

10.9 millions DOF

237 pair of contacts

46 parts

2 cores 2 cores 6-8 cores 6-8 cores

Default: Direct solver, SMP, 2 cores

Iterative solver, SMP, 8 cores Iterative solver,

SMP, 8 cores

Improvement in elapsed time and memory used:

Large, for different solver configurations (91%).

Low, for different hardwares (2%).

Best results running on one node.

Benefits:

• Removal of simplifications and idealizations.

• Increased accuracy.

• Employment of more realistic behaviors for interactions between parts.

• Availability of extra information for designing.

2 cores 16-32 cores

Iterative solver, SMP, 16 cores

17

SPEED

ACCURACY

LARGE ASSEMBLIES




18

WIND TURBINE TRANSIENT


Objective: Analyze the flow unsteady in the nacelle region due to the effect of the rotor disturbance on the oncoming wind field.

Benefits:

• Simulate 3D Transients (Blades, nacelle and tower). • Increase the number of elements in area of interest. • Turbulence model more representative of the reality.

Control the Wind Turbine in an optimal manner, in order to optimize the energy production

Statistics

N° of elements 4 Million

N° of iterations 6500

Time in 7 cores 3 weeks

Time in 32 cores 5 days

19

CFD SIMULATION OF PRESSURE FLUCTUATION ON HYDRAULIC TURBINES


• By increasing calculation power we can perform complex simulations, complying deadlines defined in the project’s design phase.

• Evaluation of the effect of vane installation in the draft tube cone on the pressure fluctuations for a rehabilitation project.

20

CALCULATION SPEED INCREASE


0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Cal

cula

tio

n S

pe

ed

Nodes Millions

Workstations

Cluster

𝑵°

𝑰𝒕𝒆

𝒓𝒂

𝒕𝒊𝒐

𝒏𝒔

𝑬𝒒

𝒖𝒂

𝒕𝒊𝒐

𝒏𝒔

∗𝑻

𝒊𝒎𝒆

∗𝑪

𝒐𝒓

𝒆𝒔

• On the range of everyday meshes (5-8 millions), Dell HPC Cluster increases the calculation speed respect to the Workstations .

• Dell HPC Cluster allows us to use meshes up to 35 millions elements.

21

NORMALIZED TIME REDUCTION


𝑻𝒊𝒎

𝒆

𝑵𝒐

𝒅𝒆

𝒔∗

𝑰𝒕𝒆

𝒓𝒂

𝒕𝒊𝒐

𝒏𝒔

1

10

100

1000

0 8 16 24 32 40

No

rmal

ize

d T

ime

N° Cores

Cluster

Workstation

22

Parallelization improves run times

BELOMONTE – ROTOR & SATOR


y = 1026x-0.519 R² = 0.9796

0 mn

50 mn

100 mn

150 mn

200 mn

250 mn

300 mn

350 mn

400 mn

0 20 40 60 80 100 120 140

Tiem

po

[m

inu

tos]

N° de Cores

HPC Cores / Nodes

7500 Cores / Nodes

Statistics

N° of elements 48 Million

N° of iterations 60

Time in 7 cores 5h50

Time in 32 cores 2h20

23

A 70% time decrease could be obtained

SPEED

ACCURACY

LARGE ASSEMBLIES




24

• Multiphysics simulations of fluid/structure interactions can be approached in two different ways :

Modeling fluid through acoustic elements or

Using a bidirectional interaction CFD and FEM simultaneously.

• Multiphysics simulations associated with acoustic phenomena, ie wave, are performed within the same scope FEM.

FLUID STRUCTURE INTERACTION - ACOUSTIC

25

NATURAL FREQUENCIES, RUNNER AND SHAFT LINE


26

STAY VANES NORMAL MODES


Total 1,168,971

Estructura 582,401

Fluido 800,770

Total 829,350

Estructura 322,077

Fluido 507,273

2,535,484

Out of Core 10 GB

Incore 50 GBTotal used

Maximum

Memory

Statistics

Number of equations

Mesh

Number of

nodes

Number of

elements

27

EVALUATION OF DYNAMIC RESPONSE AND COUPLINGS BETWEEN HYDRAULIC DUCTS AND STRUCTURAL PARTS

FLUID STRUCTURE INTERACTION - ACOUSTIC Harmonic Response

200 m

Vibrations caused by pressure fluctuations in the draft tube can be predicted

28

CONCLUSIONS

• HPC Cluster has increased the CFD calculation capability. Now bigger and more complex models can be used in design stages (complete machines, multi-phase simulations, etc.).

• Optimal hardware configurations for CFD are not the same as FEM.

• CFD computations take advantage of cores number, in a different way that FEM.

• Time grows exponentially, when FEM resolution needs to use virtual memory.

• In FEM models, run times can be optimized greatly by using correct settings according to the hardware resources, size of the model and type of analysis.

• Optimization of the elapsed time is closely related to the solver configuration. Large improvements can be achieved even when a local workstation with 2 cores is used. In some cases best results are obtained by using only one node having a maximum amount of cores.

• Licensing strategy, "HPC Pack" or "Workgroup", must be carefully designed, depending on the amount of users and available hardware resources.

29

FUTURE CHALLENGES

• Do a benchmark with GPU platforms.

• Assess the impact on performance by swapping on "SSD" instead on a "HDD".

• Setting up a specific workstation for FEM simulations, compounds by a large amount of RAM and as many cores as possible.

• Full fluid structure interaction CFD&FEM.

30

107 years…

La Tasajera

… innovating

HP Tocoma

107 years…

Bom Jardin

… providing total solutions

Agua Doce

www.impsa.com

Documents

High Performance Computer Simulations - · PDF fileHigh Performance Computer Simulations Borrajo Juan*, ... “Battle of Speed Machines”, ... CFD Calculation X8