Project Status Report€¦ · • Electra was 100% utilized with a power draw of 353 kilowatts (kW). • Approximately 2,500 gallons of water and 8 kW of power were used to cool Electra

www.nasa.gov

Project Status ReportHigh End Computing CapabilityStrategic Capabilities Assets Program

Dr. Rupak Biswas – Project ManagerNASA Ames Research Center, Moffett Field, [email protected](650) 604-4411

October 10, 2017

National Aeronautics and Space Administration High-End Computing Capability ProjectOctober 10, 2017

Electra Supercomputer Augmented with New Skylake Processors

Mission Impact: To meet NASA’s rapidly increasing requirements for high-performance computing, HECC must regularly and significantly upgrade and replace the supercomputing resources provided to the agency.

The expansion of Electra utilizes HPE’s E-cell design, which is a liquid cooled, high-density design. An E-cell consists of two 42U E-racks, which are separated by a cooling rack. The E-Cell is a sealed unit and uses a closed-loop cooling system to provide efficient heat removal.

POCs: Bob Ciotti, [email protected], (650) 604-4408, NASA Advanced Supercomputing (NAS) Division; Davin Chan, [email protected], (650) 604-3613, NAS Division, CSRA LLC

• The HECC Supercomputing Systems team augmented the Electra supercomputer with the latest generation Intel Xeon Gold 6148 (Skylake) processors.

• This expansions adds 1,152 Skylake nodes to Electra, increasing the system’s total peak performance from 1.23 to 4.78 petaflops.

• The Skylake nodes have the latest generation Extended Data Rate (EDR) InfiniBand (IB) and is interoperable with Electra’s existing Fourteen Data Rate (FDR) IB. EDR IB provides a 100 Gb/s interconnect and is backwards compatible with Electra’s existing FDR 56 Gb/s fabric.

• The augmentation will be available to the general user community in mid-October after system testing and an SBU rate calculation.

2


The Heat is on, But Electra Keeps Cool!

Mission Impact: Electra is already providing NASA its return on investment by providing a stable robust modular supercomputing facility for HECC users.

Photographs of the control screen for the Electra module; showing CPU utilization and operating conditions.

POC: Leigh Ann Tanner, [email protected], (650) 604-4468, NASA Advanced Supercomputing Division, CSRA LLC

• The Modular Supercomputing Facility was designed to use outside air to cool the Electra supercomputer.

• On September 1, 2017, at 4:34 p.m., the outside air temperature at NASA Ames reached a record high of 104°F. The MSF cooled the outside air by evaporating water into the incoming air stream.‒ Evaporative media saturated with water yielded a

25°F temperature drop.‒ Cold aisle temperature stayed within the American

Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) Operational Guidelines of 59°-81°, without mechanical refrigeration.

• Electra was 100% utilized with a power draw of 353 kilowatts (kW).

• Approximately 2,500 gallons of water and 8 kW of power were used to cool Electra on Sept.1.‒ If Electra had been in building N258, 5,000 gallons

of water and 120 kW of power to cool would have been used.

• The MSF is providing NASA users with a robust facility that is environmentally friendly, saves money in utilities, and is dependable.

3


HECC Property Management Achievements in 2017

Mission Impact: Accurate tracking of assets through their life cycle and removal of NASA data prior to system disposal ensures control of government equipment and prevents loss of NASA data.

During the annual equipment inventory, HECC property custodians account for all equipment associated with the NASA Advanced Supercomputing (NAS) facility, including all components of the Pleiades supercomputers and associated hardware.

POC: Judy Kohler, [email protected], (650) 604-4303, NASA Advanced Supercomputing Division, CSRA LLC

• Property custodians responsible for HECC/NAS property tracked more than 1,500 pieces of decaled equipment with a value of more than $128.7 million. Each piece of equipment requires multiple updates through the life cycle of its use.

• Custodians successfully completed the Annual equipment inventory three months earlier than last year’s inventory, with a 99.93% scan rate, an improvement over 2016 numbers.

• HECC/NAS custodians were interviewed by Ames Logistics department staff for best practices and innovative onsite procedures, to be shared with other Ames Custodians.

• The ongoing tracking and management of equipment during FY17 included:– RFID retagging of more than 130 pieces of

equipment.– Ames Logistics used HECC’s property waiver memo

as the best-practice example for all other groups.– Tagging 181 new pieces of equipment.– Excessing 197 pieces of equipment.– Resolution of more than 750 tickets, most relating to

equipment location and ownership updates.

4


HECC Facility Hosts Several Visitors andTours in September 2017

Christopher Buchanan, HECC networking, security, and facilities lead, gives a computer room presentation to NASA student interns.

POC: Gina Morello, [email protected], (650) 604-4462, NASA Advanced Supercomputing Division

• HECC hosted 8 tour groups in September; guests learned about the agency-wide missions being supported by HECC assets, and some groups also viewed the D-Wave 2X quantum computer system. Visitors this month included:– Richard J. Gilbrech, Director, Stennis Space Center, Ken

Human, Associate Director, and Rodney McKellip, Assistant to the Director, visited the NAS facility as part of their tour of NASA Ames.

– Navin Shenoy, Vice President and General Manager of the Data Center Group at Intel Corporation, was briefed by NAS Division management.

– Three computer science researchers from the Japan Aerospace Exploration Agency (JAXA) were given a technical review of HECC/NAS.

– Daniel Ramos, FBI supervisory special agent, Joint Terrorism Task Force (JTTF) for the San Francisco Office, accompanied by his JTTF Squad comprised of approximately six special agents and intelligence analysts who received a HECC/NAS overview, tour, and security plan brief.

– Jeff Volosin and Juan Cifuentes, members of the Transiting Exoplanet Survey Satellite (TESS) team from Goddard Space Flight Center received a HECC/NAS overview and visualization review.

– 20 new interns from the NASA Ames Intern Program.

5


Papers

• “The Stellar IMF from Isothermal MHD Turbulence,” T. Haugbolle, P. Padoan, A. Nordlund, arXiv:1709.01078 [astro-ph.GA], September 4, 2017. *https://arxiv.org/abs/1709.01078

• “The Effects of Magnetic Fields and Protostellar Feedback on Low-Mass Cluster Formation,” A. Cunningham, et al., arXiv:1709.01277 [astro-ph.GA], September 5, 2017. *https://arxiv.org/abs/1709.01277

• “Inefficient Angular Momentum Transport in Accretion Disk Boundary Layers: Angular Momentum Belt in the Boundary Layer,” M. Belyaev, E. Quataert, arXiv:1709.01197 [astro-ph.HE], September 5, 2017. *https://arxiv.org/abs/1709.01197

• “Hidden Founders? Strong Bottlenecks and Fine-Scale Genetic Structure in Mangrove Populations of the Cameroon Estuary Complex,” M. Ngeve, T. Van der Stocken, D. Menemenlis, N. Koedam, L. Triest, Hydrobiologia, September 5, 2017. *https://link.springer.com/article/10.1007/s10750-017-3369-y

• “Super-Eddington Accretion Disks Around Supermassive Black Holes,” Y.-F. Jiang, J. Stone, S. Davis, arXiv:1709.02845 [astro-ph.HE], September 8, 2017. *https://arxiv.org/abs/1709.02845

• “Dynamics of Mixed Convective–Stably-Stratified Fluids,” L.-A. Couston, D. Lecoanet, B. Favier, M. Le Bars, Physical Review Fluids, vol. 2, September 13, 2017. *https://journals.aps.org/prfluids/abstract/10.1103/PhysRevFluids.2.094804

6

*HECCprovidedsupercomputingresourcesandservicesinsupportofthiswork


Papers (cont.)

• “The Dehydration of Water Worlds via Atmospheric Losses,” C. Dong, et al., The Astrophysical Journal Letters, vol. 847, no. 1, September 14, 2017. *http://iopscience.iop.org/article/10.3847/2041-8213/aa8a60/meta

• “Large Wind Shears and Their Implications for Diffusion in Regions with Enhanced Static Stability: The Mesopause and the Tropopause,” H.-L. Liu, Journal of Geophysical Research: Atmospheres, published online September 19, 2017. *http://onlinelibrary.wiley.com/doi/10.1002/2017JD026748/full

• “An Assessment of Ground Level and Free Tropospheric Ozone Over California and Nevada,” E. Yates, et al., Journal of Geophysical Research: Atmospheres, published online September 19, 2017. *http://onlinelibrary.wiley.com/doi/10.1002/2016JD026266/full

• “Use of Cloud Radar Doppler Spectra to Evaluate Stratocumulus Drizzle Size Distributions in Large-Eddy Simulations with Size-Resolved Microphysics,” J. Rémillard, et al., Journal of Applied Meteorology and Climatology, vol. 56, no. 9, September 20, 2017. *http://journals.ametsoc.org/doi/abs/10.1175/JAMC-D-17-0100.1

• “Light Scattering by Hierarchical Aggregates,” L. Kolokolova, et al., Journal of Quantitative Spectroscopy and Radiative Transfer, September 21, 2017. *http://www.sciencedirect.com/science/article/pii/S0022407317304910

7



Papers (cont.)

• “Kinetic Alfvén Turbulence: Electron and Ion Heating by Particle-in-Cell Simulations,” R. S. Hughes, S. P. Gary, J. Wang, T. Parashar, The Astrophysical Journal Letters, vol. 847, no. 2, September 25, 2017. *http://iopscience.iop.org/article/10.3847/2041-8213/aa8b13

• “Enhancing Open National Combustion Code and Application of Energy Efficient Engine Combustor,” K. Miki, J. Moder, M.-S. Liou, Journal of Propulsion and Power, published online August 31, 2017.https://arc.aiaa.org/doi/10.2514/1.B36610

8



Presentations

• “OpenMP Doacross Loops in Practice: A Mini-Application Case Study,” G. Jost, H. Jin, OpenMPCon, Stony Brook University, September 18-20, 2017. *http://iopscience.iop.org/article/10.3847/2041-8213/aa8b13/meta

• 9th Ablation Workshop, Bozeman, MT, August 30-31, 2017– “Particle Methods for Tortuosity Factors in Porous Media” J. Ferguson– “Development of Type 3 Ablator Response Model under the ESM Project,” N. N. Mansour– “Investigation of the High-Energy Oxidation of FiberForm from DSMC Analysis of Molecular

Beam Experiments,” A. Borner– “Flow-Tube Reactor Experiments on the High Temperature Oxidation of Carbon Weaves,”

F. Panerai– “Full-Scale Mars Science Laboratory Tiled Heatshield Material Response,” J. Meurisse

9



News and Events

• NASA Supercomputing Strategy Takes the Road Less Traveled, The Next Platform, August 31, 2017—For a large institution playing at the leadership-class supercomputing level, NASA tends to do things a little differently than its national lab and academic peers. NAS Advanced Computing Branch Chief Bill Thigpen explains the benefits of taking the modular route at NASA's Ames Research Center, starting with the Electra supercomputer.https://www.nextplatform.com/2017/08/31/nasa-supercomputing-strategy-takes-road-less-traveled/

10


HECC Utilization

11

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Pleiades Endeavour Merope Electra Production

Share Limit

Job Drain

Dedtime Drain

Limits Exceeded

Unused Devel Queue

Insufficient CPUs

Held

Queue Not Schedulable

Not Schedulable

No Jobs

Dedicated

Down

Degraded

Boot

Free/Testing

Used

September 2017


HECC Utilization Normalized to 30-Day Month

12

0

4,000,000

8,000,000

12,000,000

16,000,000

20,000,000

24,000,000

28,000,000

Stan

dard

Bill

ing

Uni

ts

NAS

NLCS

NESC

SMD

HEOMD

ARMD

Alloc.toOrgs


HECC Utilization Normalized to 30-Day Month

13

0

2

4

6

8

10

12

Stan

dard

Bill

ing

Uni

ts in

Mill

ions

ARMD ARMD Allocation With Agency Reserve

ARMD

1234

56

1 16WestmereracksretiredfromPleiades2 10BroadwellracksaddedtoPleiades3 4BroadwellracksaddedtoPleiades4 14(All)Westmereracks retiredfromPleiades5 14BroadwellRacksaddedtoPleiades6 16ElectraBroadwellRacksinProduction,20Westmere

1/2racksaddedtoMerope

0

2

4

6

8

10

12

Stan

dard

Bill

ing

Uni

ts in

Mill

ions

HEOMD NESC HEOMD+NESC Allocation

HEOMD, NESC

54

321

6

0

2

4

6

8

10

12

Stan

dard

Bill

ing

Uni

ts in

Mill

ions

SMD SMD Allocation With Agency Reserve

SMD

21

3

45

6


Tape Archive Status

14

0

40

80

120

160

200

240

280

320

360

400

440

480

520

560

600

Unique File Data Unique Tape Data Total Tape Data Tape Capacity Tape Library Capacity

Peta

Byt

es

Capacity

Used

HECC

Non Mission SpecificNAS

NLCS

NESC

SMD

HEOMD

ARMD

September 2017


Tape Archive Status

15

0

40

80

120

160

200

240

280

320

360

400

440

480

520

560

600

Peta

Byt

es

Tape Library Capacity

Tape Capacity

Total Tape Data

Unique Tape Data


Pleiades:SBUs Reported, Normalized to 30-Day Month

16

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

14,000,000

16,000,000

18,000,000

20,000,000

22,000,000

Stan

dard

Bill

ing

Uni

ts

NAS

NLCS

NESC

SMD

HEOMD

ARMD

Alloc. to Orgs


Pleiades:Devel Queue Utilization

17

0

250,000

500,000

750,000

1,000,000

1,250,000

1,500,000

1,750,000

2,000,000

2,250,000

2,500,000

Stan

dard

Bill

ing

Uni

ts

NAS

NLCS

NESC

SMD

HEOMD

ARMD

Devel Queue Alloc.


Pleiades:Monthly Utilization by Job Length

18

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000

4,500,000

0 - 1 > 1 - 4 > 4 - 8 > 8 - 24 > 24 - 48 > 48 - 72 > 72 - 96 > 96 - 120 > 120

Stan

dardBillingUnits

JobRunTime(hours) September 2017


Pleiades:Monthly Utilization by Size and Mission

19

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000

4,500,000

5,000,000

5,500,000

1 - 32 33 - 64 65 - 128 129- 256 257 - 512 513-1024

1025-2048

2049-4096

4097 -8192

8193 -16384

16385 -32768

32769 -65536

Stan

dardBillingUnits

JobSize(cores)

NAS

NLCS

NESC

SMD

HEOMD

ARMD

September 2017


Pleiades:Monthly Utilization by Size and Length

20

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000

4,500,000

5,000,000

5,500,000

1 - 32 33 - 64 65 - 128 129-256

257 -512

513-1024

1025-2048

2049-4096

4097 -8192

8193 -16384

16385 -32768

32769 -65536

Stan

dardBillingUnits

JobSize(cores)

>120hours

>96- 120hours

>72- 96hours

>48- 72hours

>24- 48hours

>8- 24hours

>4- 8hours

>1- 4hours

0- 1hours

September 2017


Pleiades:Average Time to Clear All Jobs

21

0

24

48

72

96

120

144

168

192

216

240

264

288

312

Oct-16 Nov-16 Dec-16 Jan-17 Feb-17 Mar-17 Apr-17 May-17 Jun-17 Jul-17 Aug-17 Sep-17

Hours

ARMD HEOMD/NESC SMD


Pleiades:Average Expansion Factor

22

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00


ARMD HEOMD SMD


Electra:SBUs Reported, Normalized to 30-Day Month

23

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

Stan

dard

Bill

ing

Uni

ts

NAS

NLCS

NESC

SMD

HEOMD

ARMD

Alloc. to Orgs


Electra:Devel Queue Utilization

24

0

25,000

50,000

75,000

100,000

125,000

150,000

175,000

Stan

dard

Bill

ing

Uni

ts

NAS

NLCS

NESC

SMD

HEOMD

ARMD

Devel Queue Allocation


Electra:Monthly Utilization by Job Length

25

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

0 - 1 > 1 - 4 > 4 - 8 > 8 - 24 > 24 - 48 > 48 - 72 > 72 - 96 > 96 - 120 > 120

Stan

dardBillingUnits


National Aeronautics and Space Administration High-End Computing Capability ProjectOctober 10, 2017 26

Electra:Monthly Utilization by Size and Mission

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1 - 32 33 - 64 65 - 128 129- 256 257 - 512 513- 1024 1025-2048

2049-4096

4097 -8192

8193 -16384

Stan

dardBillingUnits

JobSize(cores)

NAS

NLCS

NESC

SMD

HEOMD

ARMD

September 2017


Electra:Monthly Utilization by Size and Length

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1 - 32 33 - 64 65 - 128 129- 256 257 - 512 513- 1024 1025-2048

2049-4096

4097 -8192

8193 -16384

Stan

dardBillingUnits

JobSize(cores)

>120hours

>96- 120hours

>72- 96hours

>48- 72hours

>24- 48hours

>8- 24hours

>4- 8hours

>1- 4hours

0- 1hours

September2017


Electra:Average Time to Clear All Jobs

28

0

48

96

144

192

240

288

336


Hours

ARMD HEOMD/NESC SMD


Electra:Average Expansion Factor

29

1

1.5

2

2.5

3

3.5

4

4.5

5


ARMD HEOMD SMD


Merope:SBUs Reported, Normalized to 30-Day Month

30

0

200,000

400,000

600,000

800,000

1,000,000

Stan

dard

Bill

ing

Uni

ts

NAS

NLCS

NESC

SMD

HEOMD

ARMD

Alloc. to Orgs


Merope:Monthly Utilization by Job Length

31

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

0 - 1 > 1 - 4 > 4 - 8 > 8 - 24 > 24 - 48 > 48 - 72 > 72 - 96 > 96 - 120 > 120

Stan

dardBillingUnits



Merope:Monthly Utilization by Size and Mission

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

500,000

1 - 32 33 - 64 65 - 128 129- 256 257 - 512513- 1024 1025-2048

2049-4096

4097 -8192

8193 -16384

16385 -32768

Stan

dardBillingUnits

JobSize(cores)

NAS

NLCS

NESC

SMD

HEOMD

ARMD

September 2017


Merope:Monthly Utilization by Size and Length

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

500,000

1 - 32 33 - 64 65 - 128 129- 256 257 - 512 513-1024

1025-2048

2049-4096

4097 -8192

8193 -16384

16385 -32768

Stan

dardBillingUnits

JobSize(cores)

>120hours

>96- 120hours

>72- 96hours

>48- 72hours

>24- 48hours

>8- 24hours

>4- 8hours

>1- 4hours

0- 1hours

September 2017


Merope:Average Time to Clear All Jobs

34

0

12

24

36

48

60

72


Hours

ARMD HEOMD/NESC SMD

160


Merope:Average Expansion Factor

35

1.00

2.00

3.00

4.00

5.00

6.00


ARMD HEOMD SMD


Endeavour:SBUs Reported, Normalized to 30-Day Month

36

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

100,000

Stan

dard

Bill

ing

Uni

ts

NAS

NLCS

NESC

SMD

HEOMD

ARMD

Alloc. to Orgs


Endeavour:Monthly Utilization by Job Length

37

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,000

0 - 1 > 1 - 4 > 4 - 8 > 8 - 24 > 24 - 48 > 48 - 72 > 72 - 96 > 96 - 120 > 120

Stan

dardBillingUnits



Endeavour:Monthly Utilization by Size and Mission

38

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

50,000

1 - 32 33 - 64 65 - 128 129- 256 257 - 512 513- 1024

Stan

dardBillingUnits

JobSize(cores)

NAS

NESC

SMD

HEOMD

ARMD

September 2017


Endeavour:Monthly Utilization by Size and Length

39

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

50,000

1 - 32 33 - 64 65 - 128 129- 256 257 - 512 513- 1024

Stan

dardBillingUnits

JobSize(cores)

>120hours

>96- 120hours

>72- 96hours

>48- 72hours

>24- 48hours

>8- 24hours

>4- 8hours

>1- 4hours

0- 1hours

September 2017


Endeavour:Average Time to Clear All Jobs

40

0

12

24

36

48

60

72

84

96


Hours

ARMD HEOMD/NESC SMD


Endeavour:Average Expansion Factor

41

1.00

1.50

2.00

2.50

3.00

3.50

4.00


ARMD HEOMD SMD

Documents

Project Status Report€¦ · • Electra was 100% utilized with a power draw of 353 kilowatts (kW). • Approximately 2,500 gallons of water and 8 kW of power were used to cool Electra