"Cloud Computing for HPC"

1º WCN - Workshop on Cloud Networks

Collaborative Research in Cloud Computing: future and challenges

Cloud Computing for HPC Cloud Computing for HPC

Philippe O. A. NavauxPhilippe O. A. NavauxGPPD - Informatics Institute – UFRGSGPPD - Informatics Institute – UFRGS

Grupo de Processamento Paralelo e Distribuído

1º WCN - Workshop on Cloud Networks July 06, 2016 2

High performance applicationsHigh performance applications

Applications that require a lot of computing Fluid dynamics

Computational biology

Seismic models

Climatological models BRAMS and OLAM

HPC is traditionally obtained with the use of supercomputers, clusters and grids

3

Fluids DinamicFluids Dinamic

Sterling e Brandt 2012


1997 2007DoE

Combustion EngineCombustion Engine

5

Molecular DinamicMolecular Dinamic

Sterling e Brandt 2012

6

HuricanesHuricanesDisasters PreventionDisasters Prevention

Click to edit the outline text formatSecond Outline Level

Third Outline LevelFourth Outline Level

Fifth Outline LevelSixth Outline Level

Seventh Outline LevelAzure for Research Workshop

Click to edit the outline text format Second Outline Level

Third Outline LevelFourth Outline Level

Fifth Outline LevelSixth Outline Level

Seventh Outline LevelMay 25-26, 2015

“Numerous reports have documented the technical challenges and nonviability of simply scaling Numerous reports have documented the technical challenges and nonviability of simply scaling existing computer designs to reach exascale,” said Dongarra. “Drawing from these reports and existing computer designs to reach exascale,” said Dongarra. “Drawing from these reports and experience, our subcommittee has identified the top 10 computing technology advancements that experience, our subcommittee has identified the top 10 computing technology advancements that are critical to making a productive, economically viable exascale system.” (Dongarra 2014)are critical to making a productive, economically viable exascale system.” (Dongarra 2014)

Top 10 avancemments:Top 10 avancemments:• Energy efficiency,Energy efficiency,• Interconnect technology,Interconnect technology,• Memory Technology,Memory Technology,• Scalable System Software,Scalable System Software,• Programming systemsProgramming systems

Data managementData management

Exascale Algorithms,Exascale Algorithms,

Algorithms for discovery, design, Algorithms for discovery, design, and decision,and decision,

Resilience and correctness,Resilience and correctness,

Scientic productivity.Scientic productivity.


Supercomputers TitanSupercomputers Titan


Questions ?Questions ?

Can we process applications that need high performance processing on cloud?

E Science

Did the performance suitable?


Cluster vs CloudCluster vs Cloud

Traditional hardware Cluster Hardware cost

Hardware maintenance

People for maintenance

Cloud Computing Without acquisition and maintenance cost

Minimum staff

Pay per use


Challenges in applications migration to cloudChallenges in applications migration to cloud

Portability: Lack of application and code documentation, slow down migration. Scientific applications have a very specific problem domains. Migration requires a high level knowlegde of application . Applications developed without version control or comments creates a bottleneck in migration (legacy code)

Costs Porting application code to PaaS. Number of processing instances Acceptable time to solution


Challenges in applications migration to cloudChallenges in applications migration to cloud

Resources Network

Scaling, instance migration, resilience.

Storage Local and distributed file systems.

Processing Number of instances vs number of processors.

Applications CPU-bound, IO-bound, Network-bound.


Case: Weather ForecastingCase: Weather Forecasting

Numerical Weather Forecast

Used to accurately predict atmosphericbehavior for a future time period.Represents the state of the atmosphereat a xed time (temperature,wind components, etc) over a discretizeddomain.The results are denominated weatheror climate forecasts, a few days forweather, months for climate.



BRAMS Climate ModelBRAMS Climate Model


BRAMS ModelBRAMS Model

Brazilian Regional Atmospheric Modeling System

Mesoscale model based on

RAMS, developed by the Atmospheric

Science Department at the CSU,

adapted to Brasil Climate.

The main objective of BRAMS

is to provide a single model

to Brazilian Regional Weather

Centers. Each center execute

mesoscale forecasts in their geographical area.


BRAMS is a free BRAMS is a free softwaresoftware


BRAMS Ensemble ForecastBRAMS Ensemble Forecast

Ensemble Forecast

The impact of initial condition error

propagation in the forecast integration

time can be alleviated by

the ensemble forecast technique.

A set of independent integrations

are done, with small variations

in initial conditions among each

other, resulting in distinct forecasts.


BRAMS - ECMWF MethodBRAMS - ECMWF Method 12 executions per year (for 10 years)


BRAMS Result example: Rain PrecipitationBRAMS Result example: Rain Precipitation


BRAMS in Azure

Affinity groups.Wrapped in a Cloud ServiceProcessing VMs Created on demandFrontend is the only fixed instanceBilled by forecast runService is fully automatedData stored in Azure file service


Approach:Using Azure File Srvice:

Pros:•Legacy Compatibility•Using SMB•Highly scalable, replication•Easy to setup

Cons:•No file permissions•Throughput limited•(Preview)•Metadata is lost


Shared Usage

Multiples instances of BRAMS in Cloud

72 hours

4 regions of South America

Sharing input data


Conclusions of HPC in Cloud: Climate caseConclusions of HPC in Cloud: Climate case

The main objective of the work was to migrate the Brazilian Regional

Atmospheric Modeling System (BRAMS) to a new platform that

uses the advantages provided by a cloud computing infrastructure

and evaluate the challenges of this procedure.

The goals of the work was:

Migrate Brams to Azure.

Measure the performance of the migrated version.

Analyse the cost of execution to achieve useful results.

Study the viability of usage of the implemented model by other weather centers.


Cloud and SupercomputingCloud and Supercomputing

Cloud Companies are installing Supercomputing

facilities to improve their processing power in

executing in cloud.


Conclusion of HPC in Cloud and with Supercomputing Conclusion of HPC in Cloud and with Supercomputing facilitiesfacilities

- we can use HPC applications in cloud

-.the performance is not the same

- but cloud are changing including supercomputers features

- cloud hadd VM facilities

BUT

- interconnection is one of the challenges

- I/O is another challenge

1º WCN - Workshop on Cloud Networks

Thanks!Thanks!

Collaborative Research in Cloud Computing: future and challenges

Cloud Computing for HPC Cloud Computing for HPC

Philippe O. A. NavauxPhilippe O. A. NavauxGPPD - Informatics Institute – UFRGSGPPD - Informatics Institute – UFRGS

Grupo de Processamento Paralelo e Distribuído