EXPERIMENT DESCRIPTION AND EVALUATION - CORDIS · 2017-05-11 · computation. AVL-List as independent software vendor in the experiment benefits by new customers or more sold licences

EXPERIMENT DESCRIPTI ON AND EVALUATION

Deliverable D125.1

Circulation: PU: Public Lead partner: SES-Tec Contributing partners: SES-Tec, AVL, ARCTUR, UNott Authors: Dalibor Jajcevic, Fabian Rasinger,

Tess Roper, Julie Waldron Quality Controllers: Reinhard Tatschl, Wolfgang Lang Version: 1.0 Date: 29.04.2016

D125.1 CloudFlow (FP7-2013-NMP-ICT-FoF-609100)

II

©Copyright 2013-2016: The CloudFlow Consortium

Consisting of original partners

Fraunhofer Fraunhofer Institute for Computer Graphics Research, Darmstadt, Germany

SINTEF STIFTELSEN SINTEF, Department of Applied Mathematics, Oslo, Norway

JOTNE JOTNE EPM TECHNOLOGY AS

DFKI DEUTSCHES FORSCHUNGSZENTRUM FUER KUENSTLICHE INTELLIGENZ GMBH

UNott THE UNIVERSITY OF NOTTINGHAM

CARSA CONSULTORES DE AUTOMATIZACION Y ROBOTICA S.A.

NUMECA NUMERICAL MECHANICS APPLICATIONS INTERNATIONAL SA

ITI ITI GESELLSCHAFT FUR INGENIEURTECHNISCHE INFORMATIONSVERARBEITUNG MBH

Missler Missler Software

ARCTUR ARCTUR RACUNALNISKI INZENIRING DOO

Stellba STELLBA HYDRO GMBH & CO KG

ESS EUROPEAN SENSOR SYSTEMS SA

HELIC HELIC ELLINIKA OLOKLIROMENA KYKLOMATA A.E.

ATHENA RC ATHENA RESEARCH AND INNOVATION CENTER IN INFORMATION COMMUNICATION & KNOWLEDGE TECHNOLOGIES

INT INTROSYS-INTEGRATION FOR ROBOTIC SYSTEMS-INTEGRACAO DE SISTEMAS ROBOTICOS SA

SIMPLAN SIMPLAN AG

UNI KASSEL UNIVERSITAET KASSEL

BOGE BOGE KOMPRESSOREN OTTO BOGE GMBH & CO KG

CAPVIDIA CAPVIDIA NV

SES-TEC SES-TEC OG

AVL AVL LIST GMBH

nablaDot NABLADOT SL

Biocurve BIOCURVE

UNIZAR UNIVERSIDAD DE ZARAGOZA

BTECH BARCELONA TECHNICAL CENTER SL

CSUC CONSORCI DE SERVEIS UNIVERSITARIS DE CATALUNYA

TTS TECHNOLOGY TRANSFER SYSTEM S.R.L.

FICEP FICEP S.P.A.

SUPSI SCUOLA UNIVERSITARIA PROFESSIONALE DELLA SVIZZERA ITALIANA (SUPSI)

This document may not be copied, reproduced, or modified in whole or in part for any purpose

without written permission from the CloudFlow Consortium. In addition to such written permission to

copy, reproduce, or modify this document in whole or part, an acknowledgement of the authors of

the document and all applicable portions of the copyright notice must be clearly referenced.

All rights reserved.

This document may change without notice.


III

DOCUMENT HISTORY

Version1 Issue Date Stage Content and Changes

1.0 29.04.2016 Final Final version to be submitted to PO

1 Integers correspond to submitted versions


EXECUTIVE SUMMARY

Aerated stirred reactors, the most common type of both small- and large-scale bioreactors, are used

for performing microbial fermentation or mammalian cell culture unit operations for the production

of biological therapeutics such as vaccines, hormones, proteins and antibodies. Usually, basic design

criteria have been adapted in such a way as to meet the requirements of cells. In particular, the shear

sensitivity requires consideration in impeller design, aspect ratio and aeration. Sufficient oxygen

transfer and carbon dioxide removal are very important criteria in selecting a bioreactor system.

Taking into account the process criteria, the scale-up process of bioreactors still presents a challenge

and requires detailed knowledge about diverse fields such as the mixing processes, agitation, aeration,

heat and mass transfer, etc.

Computational Fluid Dynamics (CFD) is a simulation approach that can be successfully used for the

characterization of bioreactors by evaluating process parameters. Useful processes can be obtained

using CFD already in an early development stage of the devices without the need for building-up a

prototype. Furthermore, CFD tools can be successfully used in the scale up/down process, in order to

reduce the number of prototypes and therewith production costs.

The main challenge in the calculation is the treatment of multiphase systems and long process time of

several hours which leads to long calculation time, which is not suitable for industrial application. In

order to overcome these limitations and to apply CFD simulations in the development process, a highly

optimized workflow and huge computational resources are required. For instance, an estimation of

the oxygen mass transfer coefficient for only one variant takes about one week of computational time

(using a single computer with 12 CPUs). This is mainly caused by the need to run transient simulations

up to 20 seconds. The time must be completely simulated and in addition the simulation has to run

with very small time steps caused by multiphase simulations (e.g. 0.01s).

Simulation is becoming more and more interesting for the industry in the case of comparing several

simulation variants but only if the calculation takes no more than one week. There are several

advantages of high performance computing, such as the possibility of parallel calculations using

“supercomputers” and simultaneously full cost control (costs per hour of use). Furthermore it offers

an opportunity to use high-end simulation technology in the development process without additional

fixed costs (such as licences and hardware costs). An optimized simulation workflow for the HPC/Cloud

infrastructure offers SMEs a possibility to exploit the advantage of this technology directly for their

own products, without huge investment costs and a long period of vocational adjustment.

The aim of the presented project was to adapt the virtual process for cloud-based multiphase

simulations of a bioreactor in order to perform a DoE analysis (Design of Experiments). SES-Tec, as the

end-user in the experiment, is planning to offer the obtained know-how and services to their

customers with a significantly faster, automated and proven simulation workflow using the CloudFlow

infrastructure. Obtained results during the project time will be used for demonstration, where DoE

calculations will be shown as a state-of-the-art method for the bioreactor analysis already available

for the complex multiphase simulations in pharmaceutical industry.

Economic impact

Using the Cloud-based simulation technology it is possible to carry out analysis of bioreactors with

clearly reduced costs due to reduced calculation time of parallel simulation variants and full cost

control. Through the parallel calculation of a huge number of variants in a very short calculation time

it enables the opportunity to get new customers. Addressing a customer segment defined by the


I

pharma, biopharma, automotive and general engineering industries, in a three years horizon SES-Tec

estimates to face a market size of around 4.7$ billion, with a potential share reaching 20.000-

50.000€/a and leading to an increase of 15% of actual turnover due to only Cloud computing.

Furthermore, SES-Tec is planning to hire a new employee, which deals exclusively with cloud

computation. AVL-List as independent software vendor in the experiment benefits by new customers

or more sold licences. Moreover, a new Cloud-based business model was created as well. In the three

years perspective, AVL is expecting an increase of the sold Cloud-based licences by 5-10% (about

150.000€ in three year perspective). Only for the Cloud computing, AVL is expecting to offer between

2 and 5 new job positions. Furthermore, AVL also expects to gain new clients and to increase the

number of AVL-FIRE users up to 15% in three-year perspective.

Technical impact

The computational time for design of experiments (DoE) analysis was decreased from five weeks to

one week. Due to huge computational resources in the cloud, all 25 simulation variants can be run in

parallel and not one after the other. This number of variants are typical for DoE analysis, but are not

limited anymore thanks to Cloud-based technology. Furthermore, the number of simulation variants

is no longer related to the in-house hardware resources and therefore no investments are needed.

Finally, each bioreactor manufacturer can benefit from proven and validated simulation technology

and workflows for this kind of application.


TABLE OF CONTENTS

Executive summary ...................................................................................................................................

1 Description of the current (engineering and manufacturing) process (PU) .................................. 1

2 Description of the (engineering and manufacturing) process based on Cloud services (PU) ........ 2

Workflow using the CloudFlow Portal ................................................................................................ 2

Bioreactor example ............................................................................................................................. 3

CFD setup files ..................................................................................................................................... 4

GridWorker ......................................................................................................................................... 6

Parameter Study ................................................................................................................................. 8

CFD results .......................................................................................................................................... 9

Checkpoint in the CF Portal............................................................................................................... 11

Storage in the CF Infrastructure ........................................................................................................ 12

Outputs in the CF Infrastructure ....................................................................................................... 12

2D and 3D post-processing ............................................................................................................... 12

Design of Experiment Analysis .......................................................................................................... 15

3 Lessons learned (PU) ..................................................................................................................... 18

4 Impact (PU) ................................................................................................................................... 19

5 Business Model (CO) ..................................................................................................................... 20

6 Execution of the Experiment (CO) ................................................................................................ 26

7 Recommendation to the CloudFlow infrastructure (CO) .............................................................. 30

8 Confidential information (CO) ....................................................................................................... 31

9 Involved Organisations ................................................................................................................. 32

10 Evaluation Details ..................................................................................................................... 33

Process overview .......................................................................................................................... 33

Issue 125.1: Cancelling a simulation in progress .......................................................................... 34

Issue 125.2: Management of Resources from End-User Perspective ........................................... 34

Appendix 1: bash script (map.sh.generic) ............................................................................................. 35

Appendix 2: User requirements and how they are met ....................................................................... 38

Appendix 3: Usability Evaluation .......................................................................................................... 41

Appendix 4: Business Model exploration concept................................................................................ 41

Appendix 5: CUSTOMER DEVELOPMENT QUESTIONNAIRE .................................................................. 44


1

1 DESCRIPTION OF THE CURRENT (ENGINEERING AND MANUFACTURING) PROCESS (PU)

SES-Tec (standing for Scientific & Engineering Simulation – Technology) supports customers in the field

of multi-physics simulations and the creation of simulation methods. The simulation services are

offered to the companies, which do not use simulation technology in-house and/or which would like

to perform more complex and time-consuming calculations, such as bioreactors. Usually SES-Tec

customers want to optimize an already existing product, develop a new one or has a concrete problem

definition, where they need SES-Tec competencies and services.

At the beginning of a simulation project process the customer discusses the technical problem with

SES-Tec. The problem usually consists of a technical issue based on fluid dynamics and

thermodynamics within a certain device. In the case of a bioreactor simulation, scale-up or –down are

the technical issues, which has to be investigated. SES-Tec analyses the issues and starts problem

analysis using simulation technology. The simulation process consists of four different steps:

preparing, pre-processing, solving and post-processing. Figure 1 shows the current workflow, which is

described in more detail in the following section.

FIGURE 1: WORKFLOW FOR A SIMULATION RUN USING INHOUSE RESOURCES

In the first step the geometry is either already available in a 3D CAD format such as .STEP files or this

is done in-house. In the second step, the geometry is pre-processed and transformed into the surface

mesh format (.STL). The 3D surface model can be imported into the AVL-FIRE® pre-processing tool and

all face selections, which are required for the meshing process and definition of the boundary

conditions, are generated. The face selections are following surfaces: impeller, sparger, air inlet

surface, bafflers, and vessel. Using the AVL-FIRE® mesh generator, the computational mesh is created.

Additionally, the CFD setup files are prepared consisting of typical bioreactor use case initial

conditions, fluid properties, rotating speed of the impeller, aeration rate, etc. After this step the input

data is ready to be transferred to the in-house calculation server. This is done via FTP protocol through

the local area network LAN (Figure 1). The calculation server is accessed either directly or via remote

desktop connection. At this point the pre-processing step is finished, which takes about a day

depending on the geometry complexity. Additional time is needed if the geometry has to be create by

SES-Tec. The calculation of the CFD simulation is executed and can take - depending on the problem

size - about one week for one bioreactor operating point to be finished. The results of the simulation


2

are visualized via remote desktop connection or they are downloaded to the local machine to be

visualized and analysed there with the corresponding desktop application, the visualisation/post-

processing tool. In the post-processing step, cuts are made through the 3D result data set to get more

insight of the physics behaviour inside the simulation domain (volume). Afterwards the results are

evaluated and summarized in order to be presented to the customer. The whole post-processing takes

around two days depending on the complexity. If there is a need of a design of experiment study, one

has to multiply the effort for one post-processing with the total number of variants.

Basic resources include the local machine, the calculation server and the CFD software licenses, which

is in our case AVL-FIRE®. Additionally, there is expertise needed to install, maintain and run the server

as well as all the software packages. The above described simulation process is identical for almost all

applications. Nevertheless, some specific steps for the bioreactor simulation are described in the next

chapters.

2 DESCRIPTION OF THE (ENGINEERING AND MANUFACTURING) PROCESS BASED ON CLOUD SERVICES (PU)

In the following sub-sections, the Cloud-based process of a bioreactor simulation is described in detail.

Workflow using the CloudFlow Portal

The workflow for the end-user using the CloudFlow Portal (CFP) is presented in Figure 2. The end-user

prepares the simulation project file on their local machine. The preparation includes the generation

of the computational mesh and the preparation of the simulation setup files. Afterwards the data can

be transferred to the SWIFT storage system using the upload client within a browser in the CloudFlow

Portal. The SWIFT storage system is a cloud-based storage solution. It is ideal for storing unstructured

data that can grow without bound.

CloudFlow’s GridWorker (GW) is executed with a right click on the “configuration” file and then

choosing the “start workflow” “GridWorker” option. The GW simulation overview opens automatically

and shows the status of the simulations. During the simulation, basic data is updated regularly and

stored. This data can be used to monitor the simulation during simulation time. When the simulation

is finished, the data can be analysed using 2D or 3D post-processing within the CloudFlow Portal or by

downloading via HTTPS and analysing the data on a local machine.

In contrast to the current in-house process by SES-Tec, in the Cloud based process the simulation step

“solving” is run on the HPC architecture by ARCTUR, which presents the main difference. Further, the

“post-processing” step can be done either in the Cloud or localy after data download.


3

FIGURE 2: WORKFLOW FOR A SIMULATION RUN IN THE CLOUDFLOW PORTAL

Bioreactor example

The example used in the project is a typical batch type bioreactor configuration, with a commonly

used Rushton impeller. It is working in a wide range of operating conditions. Required data for the

simulation and validation, such as bioreactor design, operating conditions and measurement, are

taken from the PhD Thesis of Sandadi Sandeepa, “Mass transfer, mixing, Chinese hamster ovary cell

growth and antibody production characterization using Rushton turbine and marine impellers”. In the

thesis, all relevant geometry, process data and measurement data are published. This type of

bioreactor represents a “reference” case and its widely used for the simulation validation.

Furthermore, using already published data, the confidential issues are overcome. This means, that the

obtained results within this project can be further used for the dissemination activities without any

limitations. By using the published data, a 3D model was generated as seen in Figure 3.


4

FIGURE 3: BIOREACTOR DESIGN

The bioreactor consists of a vessel, in which chemical reactions are carried out and an impeller, which

provides a good mixing of the reacting components. The baffles are responsible for the improvement

of the mixing performances, which can be found close to the walls. Finally, the sparger, which is a

drilled tube, is used for the aeration and input of bubbling gas into the culture medium.

CFD setup files

The CFD setup files are prepared using the AVL-FIRE® pre-processing tool. The 3D model can be

imported into the AVL pre-processing tool and all face selections are generated, which are required

for the meshing process and the definition of the boundary conditions. Using the AVL-FIRE® mesh

generator, the computational mesh is generated, which consists of two parts. The first one is the rotor,

which is a cylindrical cell selection (a volume selection) close to the rotation part of the impeller. The

second one is the stator, a non-moving part, as seen in Figure 4. One additional volume selection is

required for the initialization of the fill level of the medium. In the case of an investigation of different

fill levels, several volume selections have to be already prepared in the pre-processing phase. The

presented volume selection, the fill level of the medium was selected in such a way, that the volume

of the culture medium is 15 litres corresponding to the reference of the PhD thesis.

After the mesh generation process, the setup of the CFD simulation file “Case.ssf” is performed. This

step will be done in the pre-processor of AVL-FIRE®. The input data consists of initial condition, fluid

properties, rotating speed of the impeller, aeration rate, etc.


5

FIGURE 4: COMPUTATIONAL MESH

The CFD simulation setup is a multiphase setup, where two phases are considered, liquid and gas. The

liquid phase is set as water, due to the fact that almost all culture mediums have very similar material

properties to those of water. The coupling between the two phases occurs via a drag-force model,

whereas a bubble diameter has to be defined. The bubble diameter was set to 1mm and kept constant.

The rotation of the impeller was modelled using a Multiple Reference Frame (MRF) model. This is

called a “frozen rotor” technology, because the rotating mesh remains on a fixed position during the

computation. In the volume cell selection or the rotor (Figure 4), two additional forces (centrifugal

and Coriolis force) are set as source terms, which are acting on the fluid and gas. For the simulation,

the pressure-based solver, with an implicit formulation for unsteady flow, is used. For the coupling of

pressure and velocity, the SIMPLE scheme has been chosen. The k-zeta-f turbulence model was used

to simulate turbulent flow. Further, the standard wall function for near wall modelling of the turbulent

boundary layer was selected.

FIGURE 5: AVL-FIRE® PROJECT FILE STRUCTURE

The AVL-FIRE® project file <project name> consists of two folders (calculation and meshes) and the

“<name>.fpr” project file (where, “.fpr” is the AVL-FIRE® project filename extension). Inside the

calculation folder, the 3D simulation results are saved and they can be separated by different folders

for each simulation variant, e.g. Case, Case_2…, shown in Figure 5. Inside the folder “Case”, the file

“Case.ssf” (Solver Steering File) is saved, which contains all simulation input data, such as boundary


6

and start conditions, numerical schemes, convergence criteria, output data, etc. The computational

mesh is stored into the Meshes folder, namely <mesh_file_name>.flm.

When the simulation setup process is finished, the user has to do some modifications. After that, the

user has to compress the AVL-FIRE® project file to “inputs.jar” and upload it to the CloudFlow Portal.

The modifications contain the generation of a generic “Case.ssf” file and a bash script for mapping,

which contains the pre-processing, the main execution of the simulation and the post-processing. This

is explained further in the following sub-sections.

GridWorker

The GridWorker (GW) software is developed by Fraunhofer and presents a tool, which takes control

over the simulation runs and the job management. The advantage of using only one simulation setup

is that a parameter study can be run easily, without using an additional user handcraft. The GW can

be seen as a link between the user-interface, which is a web-based application and AVL-FIRE®. GW

requires a clear data structure, which is shown in Figure 6

FIGURE 6: GRIDWORKER DATA STRUCTURE

Inside the project folder, there are two different java archives (the “inputs.jar” and the “mapper.jar”

archive) and two text files (configuration and parameters). In the archive “inputs.jar” there are two

AVL-FIRE® folders, namely “Calculation” and “Meshes”, as well as one AVL-FIRE® project file “BR1.fpr”.

In the “Calculation” folder there is the “Case.ssf.generic”. It is the AVL-FIRE® input file (Solver Steering

File), which contains all the simulation input data as explained before. The “.generic” extension means

that GW will automatically replace predefined variables in case of a parameter study, which is

explained in the following chapter. The generic file is specifically prepared for the bioreactor

experiment and does not need to be modified by the end-user. The folder “Meshes” contains the

mesh file “<mesh-file-name>.flm”. The folder structure within the “inputs.jar” is similar to the AVL-

FIRE® project structure as described in the previous chapter.

The second archive “mapper.jar” contains the “map.sh.generic” bash-script, in which one can find all

required commands and input data in order to run the AVL-FIRE® calculations as well as to ensure the

data management on the HPC environment, see Appendix 1.The “map.sh.generic” file is comprises of

several parts. The first part includes the data for the GW itself. The log() function is logging the events

during the simulation process. The simulate() function consists of the linking process of “Case.fla” into

the checkpoints folder and the running commands of AVL-FIRE® for the bioreactor simulation. The

check() function is updating the files in the checkpointing folder and transferring regularly to the

SWIFT server, which can be accessed by the user. The abort() function is stopping the simulation

without any data loss. The summarize() function is post-processing the data after the simulation and

is moving it into the storage folder, on the Swift server. And the map() function consists of the

simulate() and the summarize() function, which is the standard routine of the “map.sh.generic” file.


7

At the end of the simulation the summarized data is saved in an output folder as final result. This

folder is compressed into a java archive and also stored on the Swift server, which can be accessed via

the CloudFlow Portal.The “configuration” and “parameter” files are documents, which contain the

input data for a parameter study using GW. The “configuration” file has the following input data, which

are crucial for the correct execution of a GW task:

gridworker.worker = ssh:pbs:local

gridworker.mapper = Script

gridworker.inputs = BioReactor

gridworker.project = BioReactor_02

gridworker.experiment = Simulation_08

gridworker.jobs.number = 1

gridworker.job.threads.number = 1

gridworker.task.threads.number = 12

gridworker.task.nodes.number = 1

gridworker.retries = 0

gridworker.task.debug.level = 9

gridworker.job.debug.level = 9

gridworker.worker.debug.level = 9

gridworker.jvm.stacksize.min = 2048m

gridworker.jvm.stacksize.max = 4096m

gridworker.worker.ssh.hosts = 91.223.115.4

gridworker.worker.ssh.ports = 5892

gridworker.worker.pbs.jobs.number = 4

gridworker.worker.ssh.key.name = gridworker-key-arctur-hpc-02

In this “configuration” file GW is using four jobs in the HPC cluster with 12 CPUs for each job.

All files are uploaded on the Swift server into a folder <project name> via the CloudFlow Portal. GW is

started with a right click on the “configuration” file (Figure 7).

FIGURE 7: STARTING PROCESS OF THE GRIDWORKER

After that, a new tab in the browser opens displaying the GW simulation status overview of all variants

started in this workflow. A screenshot of this tab is presented in Figure 8. This overview is used to see

the number of running, completed, failed and aborted jobs of a certain session. At the end of the

calculation the results are stored in the session folder under a random session-id name.


8

FIGURE 8: GRIDWORKER SIMULATION STATUS OVERVIEW OF ALL VARIANTS DURING SIMULATION TIME

Parameter Study

As explained in section 1.5, GW is a tool, which takes control over the job runs and can be used to

carry out a parameter study. For the purpose of setting up the parameter study, the AVL-FIRE® input

file will be modified and saved as “Case.ssf.generic”. GW identifies the “.generic” file and searches

inside the content of the file, which variables have to be replaced. The variable values are defined in

the “parameters” file, as follows:

time-step = {0.01 }

end-time = { 0.1 }

max-number-iterations = { 50 }

cpus = { 12 }

inlet-velocity = { 0.01903 0.07613 0.14 0.571 1.0 }

speed-rpm = { 95 183 300 }

For the bioreactor example, the inlet velocity of air was selected as the variable, on which the

parameter study will be performed. The value of the velocity in the “Case.ssf” file was replaced by

“@inlet-velocity@” and the file was saved with the “.generic” extension. The inlet velocities are

defined in the “parameters” file and show the following form:

“inlet-velocity = { 0.019 0.076 …}”.

Additionally the agitation rate was varied as follows:

“speed-rpm = { 95 183 300 }”.

GW now identifies five times and three parameters and then runs 15 different simulations. For each

simulation GW will automatically generate the AVL-FIRE® “Case.ssf” file by using the defined velocity

as well as agitation rate value. To make the simulation more flexible there are important simulation


9

parameters on the list for the use-case, such as time-step, end-time of simulation, maximum number

of iterations and number of CPUs per simulation.

At the end of each simulation, the predefined results of the simulation will be automatically saved by

GW, see “map.sh” bash-script described in the previous chapter. The result folders are named from

“0”, “1”, … , “15” and they can be clearly identified using the “index” file. This file consists of a list with

all parameter variations corresponding to the folder “0” – “15”. The same structure is applied to the

files in the “outputs.jar” archive, in which the data is compressed and reduced. For the bioreactor

prototype the evaluated data consists of the mass transfer coefficient and the shear-rate as a function

of time. In both values the volume data is averaged over the whole computational domain. Further

there is a “logs.jar” in the session-id folder, which contains of the log file of the GW calculation.

Additionally, there is a “results.jar” file containing further compressed data such as duplicates of

“index.dat”, parameters, etc.

CFD results

Only the checkpoint folder is created during the simulation. All other files are created after the

simulation including checkpoints and storage folder as well as index, “logs.jar”, “outputs.jar”,

“results.jar” and a summary file. The file structure is shown in Figure 9. The session folder is either

created after reaching the first checkpoint or the calculation is finished.

FIGURE 9: FILE STRUCTURE OF THE SESSION AFTER THE SIMULATION RESULTS ARE CALCULATED

The simulation is running until a steady-state condition is reached. For the parameter study, 15

different operating conditions were simulated, which included five different aeration rates (0.75, 3,

5.5, 22.5 and 39.5 sLPM, unit “standard litre per minute”).

Additionally, a variation of three different agitation rates (95, 183, 300 rpm, unit revolutions per

minute) was performed. Two volume-averaged process parameters have been evaluated from the

results. The first one is the oxygen mass transfer coefficient in culture medium and the second one is

the shear rate. The simulations have only been running for 20 seconds of process time when the two

process parameters have achieved a quasi-steady state condition.

Using the HPC environment with 12 CPUs each with a speed of 2.67 GHz the calculation takes around

13 days in real time per simulation variant. This duration mainly depends on the time step, which is

0.01, and the number of iterations per time step, which are required to reach convergence criteria. At

the end of the simulation, 3D results are written in the CGNS file format. By using the data, the flow

behaviour inside the bioreactor can be visualized using the 3D post-processing tool embedded into


10

the CloudFlow Portal. Alternatively, the data can be downloaded on the local machine and the flow

behaviour inside the bioreactor can be visualized by the AVL-FIRE® post-processing tool.

FIGURE 10: 3D SIMULATION RESULTS, AERATION RATE 3.0 [SLPM] AND AGITATION RATE 95 [RPM]

In Figure 10, air/liquid volume fraction inside the bioreactor can be seen. The red colour represents

pure water, while blue is pure air. Typically for the system one can see an air column very close to the

impeller. Between the sparger and the first impeller, a big air bubble is predicted by the simulation.

This is also a typical behaviour at higher aeration rates.

In order to validate the CFD simulation, the obtained results were compared with the measurement

results published in the PhD Thesis of Sandadi Sandeepa, “Mass transfer, mixing, Chinese hamster

ovary cell growth and antibody production characterization using Rushton turbine and marine

impellers”. The author has published oxygen mass transfer coefficient for two different operating

points, namely sparging rates. Figure 11 shows a comparison between simulation results obtained in

this experiment and measurement data published by Sandadi Sandeepa. The author has made for

each operating point three iterations. Presented simulation results show a very good agreement with

the measurement data. Due to increasing sparging rate, the oxygen mass transfer coefficient increases

also with the expected difference.


11

FIGURE 11: SIMULATION VS. MEASUREMENT

Checkpoint in the CF Portal

Checkpoint is used to view simulation data during simulation time, when final results are not

generated yet. In CFD simulation residuals are monitored continuously to check the convergence of

the simulation and secure the correctness of the final results. For this purpose, Fraunhofer has

implemented a fixed checkpointing interval, after which some simulation data is transferred to the

Swift server in order to be accessed by the user via the CloudFlow Portal. The following additional lines

are added to the GridWorker configuration file:

gridworker.job.monitoring = on

gridworker.monitor.host = 172.29.129.206

gridworker.monitor.port = 18080

gridworker.task.checkpointing = on

gridworker.task.checkpointing.interval = 03:00:00

gridworker.task.checkpointing.storage = swift

gridworker.task.checkpointing.mode = aggregate

In this case, the checkpointing interval is set to three hours, but it can be adjusted to the user’s needs.

In the bioreactor simulation the following files are monitored: “checkpoint.log”, “residuals.dat” and

“results.dat”. The file “checkpoint.log” contains all basic information of the simulation on the cloud,

which is identical to the “Case.fla” file.

In the file “results.dat” the oxygen mass transfer rate and the shear rate is calculated and listed as a

function of time. It is important to see if the two rates reached a steady state condition or not. The

residuals are checked to see if the solution of the simulation is converging or not. If the solution is

convergent and the physical values reach a steady state point the evaluation for the design of

experiment is started.

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

5

0 0,5 1 1,5 2 2,5 3 3,5

kLa

Sparging rate [slpm]

Simulation vs. Measurement

Measurement - Sandadi Sandeepa

Simulation - AVL-FIRE


12

Storage in the CF Infrastructure

The storage folder is appearing in the corresponding <task-id> folder, which is located in the session

folder after a GW simulation has finished. It is used to directly access simulation data after the

simulation and visualize it in the 2D or 3D post-processing step within the CloudFlow Portal. The

simulation data is neither reduced nor compressed in archives. The following additional lines are

written in the “configuration” file to secure correct storing of result files after the simulation.

cloud.storage.user = cloudflow

cloud.storage.tenant = cloudflow

cloud.storage.provider = openstack-swift

cloud.storage.endpoint = https://openstack.arctur.si/keystone/v2.0

cloud.storage.credential = soo4kaethaiPh9Fa

cloud.storage.password = changeit

cloud.storage.region = regionOne

Outputs in the CF Infrastructure

In addition to the storage folder there is the outputs folder, which is used for reducing and

compressing results of the CFD simulation. For this reason, files in the outputs folder are compressed

in java archives and named from “0.jar”, “1.jar”, … , “15.jar”.

Afterwards all these files are compressed to “outputs.jar”, which can be downloaded from the

CloudFlow Portal. In order to avoid handling large amounts of data, it is convenient to reduce the

results to the most important data and store it in the outputs folder. There it gets compressed

automatically and packed as previously described.

2D and 3D post-processing

In order to execute a post-processing step, one can open a file browser workflow in the CloudFlow

Portal. The 2D post-processing consists of two different charts. The first one contains the oxygen mass

transfer and the shear rate as a function of time. The second chart displays the residuals of velocity

and mass as a function of time. These two 2D charts can be generated after the simulation time via

right mouse click on the “results.dat” file and the “residuals.dat” file, respectively, and then starting a

workflow called “2D Chart”, which is illustrated in Figure 12.

FIGURE 12: 2D CHART WORKFLOW USING THE “RESIDUALS.DAT” VIA RIGHT CLICK


13

After clicking on the “2D Chart” option using the “residuals.dat” as a starting point, the 2D chart (Figure

13) is generated, displaying the residuals for some steps of a bioreactor simulation.

FIGURE 13: RESIDUALS OF VELOCITY AND MASS AS A FUNCTION OF TIME DISPLAYED IN A 2D CHART

Figure 13 shows that the solution of the simulation is converging during every time step of process

time. Additionally, the shear rate and the oxygen mass transfer for an agitation rate of 183 rpm and

an inlet velocity of 0.14 m/s are plotted in Figure 14.

Both values reached a steady-state condition. For this chart, one can start a “2D Chart” workflow via

right click on the “results.dat” file. At higher agitation rates, it takes a longer process time in the

simulation to reach a steady-state point. Although the oxygen mass transfer is still oscillating, this can

be averaged out in the final data evaluation. For this reason, the kLa-value and the shear rate are

averaged over the last two seconds and the average values are used for the design of experiment

calculation.


14

FIGURE 14: SHEAR RATE AND OXYGEN MASS TRANSFER AS A FUNCTION OF TIME FOR A FULL BIOREACTOR

SIMULATION

3D post-processing can be launched by right-clicking on the BioreaktorVB.cgns file and then executing

a workflow called “Remote Post-Processing” (Figure 15).

FIGURE 15: 3D POST-PROCESSING USING THE BIOREAKTORVB.CGNS

This opens a new browser tab. An interactive window appears and after clicking on “connect”, the 3D

simulation results for the bioreactor are displayed. These results contain the pressure, the velocity in

all three directions, the volume fraction of the fluid, the shear rate and the oxygen mass transfer rate

for every position within the bioreactor. In the standard view, the results are plotted on the surface

of the bioreactor volume. Additionally, a cut plane can be activated to see the physical behaviour of

the mixing process inside the bioreactor. A cut through the bioreactor can be made by setting the

switch “Cutting Plane” to “on”. The position and the direction of the plane can be chosen by the user.

Figure 16 shows the 3D post-processing tool, in which a cut through the pressure distribution within

a bioreactor is presented. In this manner, the 3D simulation results can be accessed via the CloudFlow

Portal by the end user using just a web browser. Otherwise, the user can download the results and

can evaluate it locally using AVL-FIRE© post-processing tool, see Figure 10.


15

FIGURE 16: 3D POST-PROCESSING TOOL DISPLAYING A CUT THROUGH THE BIOREACTOR

Design of Experiment Analysis

A Design of Experiment (DoE) was conducted in order to find out the dependencies of the agitation

speed and the sparging rate on the oxygen mass transfer rate as well as the shear rate. For this

purpose, the parameter file is extended by three agitation rate values as well as additional inlet-

velocities and it is displayed in complete form, as follows:

time-step = { 0.01 }

end-time = { 20 }

max-number-iterations = { 50 }

cpus = { 12 }

speed-rpm = { 95 183 300 }

inlet-velocity = { 0.01903 0.07613 0.14 0.0571 1.0 }

The DoE analysis is done by varying the agitation rate and the sparging rate, to find out the main

influences of these variables on the oxygen mass transfer rate and the shear rate. The sparging rate is

directly proportional to the inlet velocity, which is a boundary condition value for the bioreactor

simulation. The oxygen transfer rate is directly linked to the kLa-value calculated from the CFD results.

The simulation results are downloaded and the DoE was performed on the local machine. The

“results.dat” file of every variant was analysed and the kLa-values as well as the shear rates of the last

two seconds of process time within the simulation was averaged. This steady state values are used for

the DoE analysis. For this purpose the main effect plots (Figure 17 and Figure 18), the interaction plots

(Figure 19 and Figure 20) and the contour plots (Figure 21 and Figure 22) are created.

The main effect shows that a higher agitation rates and sparging rates lead to increasing oxygen mass

transfer rates (Figure 17). In contrast, the shear rate depends on the agitation rate (Figure 18).


16

FIGURE 17: MAIN EFFECT PLOT FOR OXYGEN MASS

TRANSFER RATE

FIGURE 18: MAIN EFFECT PLOT FOR SHEAR RATE

Regarding the interaction plots of the oxygen mass transfer rate (Figure 19), oxygen mass transfer

coefficient increases with increasing agitation and sparging rates. By sparging rates higher than 1,52

[vvm] it can be seen that the oxygen mass transfer coefficient does not linearly increase anymore.

With 183 rpm a lower value than with 300 rpm is visible. In Figure 20 with 183 rpm the shear rate is

not dependent on the sparging rates, in contrast to 95 and 300 rpms.

FIGURE 19: INTERACTION PLOT FOR OXYGEN MASS

TRANSFER RATE

FIGURE 20: INTERACTION PLOT FOR SHEAR RATE

The contour plot of the oxygen mass transfer rate and the shear rate gives an overview over the

calculated results (Figure 21 and Figure 22). These plots are important to find the optimal operation

point of the bioreactor. There is an upper limit in the agitation rate because high agitation rates lead

to high shear rates (Figure 22), which is the main reason for cell destruction in bioreactors.

FIGURE 21: CONTOUR PLOT FOR OXYGEN MASS

TRANSFER RATE

FIGURE 22: CONTOUR PLOT FOR SHEAR RATE


17

The DoE leads to a better understanding of the influences of the process parameters such as agitation

rate and sparging rate on the oxygen mass transfer and shear rate. Furthermore, this is an example

case with an automatic workflow, which now can be quickly applied to other bioreactor geometries.


18

3 LESSONS LEARNED (PU)

Cloud computing based on HPC infrastructure offers high computational resources. This is particularly

important for SMEs, who do not want to invest in their own HPC infrastructure or do not have the

financial resources to do so. There are many reasons for this; some SMEs do not want to install and

maintain HPC infrastructure themselves, or some SMEs only need high computational resources for a

short period. Furthermore, it is very time consuming and requires big effort to build an HPC cluster.

This may not be worthwhile if the resources are not in continuous use.

There are a large number of potential applications for CFD simulations and this is continuously

increasing. At the beginning, the setting up of a cloud-based service is more time consuming than

running the simulation on the local machine. This is mainly due to the effort of getting to know the

HPC infrastructure, for writing bash scripts for the data management on the cloud and for data

uploading to as well as downloading from the cloud. Especially high potential for cloud computing is

seen in standardized simulations, which have the same workflow every time with just slightly changing

boundary conditions, for instance. In this case, the bash scripts have to be only written in the

developing phase of the cloud-based workflow. After the developing phase, the unexperienced user

can use the workflow without any modifications. During the experiment, SES-Tec gained valuable

experience in cloud computing.

In the bioreactor use case a variation of two process parameters were performed, namely the sparging

rate and the agitation rate. For a detailed analysis, one needs up to 25 simulation variants, which

corresponds to a high computational effort. The GW software tool was used to simulate several

variants at the same time. For this purpose, an HPC infrastructure is necessary for a quick simulation

of the bioreactor geometry – especially in the design of experiment analysis.

Interoperability between SMEs and their customers can be enhanced, when simulation results are

accessible from the internet via the CloudFlow Portal. As CFD simulation results usually contain a large

amount of data, it is time-consuming transferring them via the internet. Especially 2D and 3D post-

processing are important tools for interoperability in an engineering process. The flexibility is

increasing, as high computational resources are just accessed, if they are needed – with full cost

control.

In addition, different use-cases for simulation applications can be offered as a cloud-service.

Depending on the software used, it can be easier or more difficult to deploy a simulation application

as cloud-service. Fortunately, AVL-FIRE® has a very open software code structure based on setup files

written in clear text. So every simulation which runs in AVL-FIRE® on a local machine can also be

offered as cloud-service. This is very useful for long process time within the simulation and many

different variants.


19

4 IMPACT (PU)

End User (SES-Tec): A proven and validated simulation technology and workflow for this kind of application is a clear benefit for SES-Tec, which prefers to use AVL-software and a HPC/Cloud infrastructure by ARCTUR. The workflow is developed so that the users input is reduced to a minimum and therewith the effort. The reduction of the costs is mainly related to the investment and service costs of the HPC infrastructure and the software. Especially for SMEs like SES-Tec, which are using HPC infrastructure on demand, this benefit is an essential one. Beside the reduction of costs, reducing calculation time is significant. This is related to the massive process parallelization ability. For instance, an analysis of huge number of simulation variants, which are usually required for a DoE analysis, can be run simultaneously, so that instead of only one simulation, an optimization process can be performed in the same time. As there is a use-case implemented into the CloudFlow infrastructure, SES-Tec now has the ability and the expertise to run time-consuming bioreactor simulation on an HPC cluster more efficiently. Having the expertise to operate a simulation in an HPC infrastructure increases the flexibility of the computational resources enormously. Whenever HPC performance is needed, it can be purchased and accessed quickly. General, the main advantage of the simulation technology is the estimation of process data, which cannot be obtained by measurement technics. An analysis without the need for building-up a prototype clearly reduces the development costs. By the use of massive parallelization, process optimization can be performed, which can directly improve the product quality, e.g. higher cell density at the end of production process. This is a new kind of service (product optimization using DoE analysis), which SES-Tec is planning to offer their customers. SES-Tec is expecting about five new customers solely due to the decrease of simulation time in the next three years. Currently, about 15% of SES-Tec turnover is obtained from the bioreactor simulation. An increase up to 30% is expected. Assuming a positive feedback from the market and new project in this area, SES-Tec would like to employ a new employer for Cloud based simulation. Beside development and prototype departments in companies, introduction of a simulation department is becoming state-of-the-art in almost all big companies and it offers new job opportunities. It is expected, that simulation technology becomes a state-of-the-art development tool in the SMEs R&D departments and the Cloud based simulations will be a perfect opportunity regarding to investment costs. During this project, SES-Tec has used the opportunity to perform a DoE analysis and produce valuable results for upcoming projects. The results will be used as demonstration to show potential customers the quality and workflow of a successfully finished project in the field of pharmaceutical industry. ISV (AVL List GmbH): Regarding to product innovation/improvement, AVL had the opportunity to adapt and validate physical models available in AVL FIRE®, such as e.g. for multi-phase treatment, species transport, etc. for the application of multi-phase flow simulation in a bioreactor. Current and potential AVL-FIRE user will get described workflow for this kind of application using an HPC/Cloud infrastructure. The workflow can be very easy adapted to any new application and AVL-FIRE simulation tool. It can be expected that the availability of the fully elaborated bioreactor experiment in the cloud based on AVL FIRE® will largely contribute to lowering the threshold for SMEs that have not yet used simulation tools at all. Cloudification of AVL-FIRE opens new market areas in especial form the SME sectors, which do not have enough computational resources in house. Thus, it can be expected increasing number of sold licences due to Cloud based business model. AVL is expecting an increase of the sold Cloud-based licences by 5-10%. Only for the Cloud computing, AVL is expecting to offer between 2 and 5 new job


20

positions. AVL also expects to gain new clients and to increase the number of AVL-FIRE® users up to 15% in three-year perspective. Furthermore, due to sophisticated models and features enables new distribution possibilities such as e.g. pharmaceutical process engineering, energy and environmental engineering, aerospace and civil engineering, etc.

CloudFlow Competence Centre CFCC: The CloudFlow Competence Centre (CFCC) can extend the HPC capacity, as they are gaining new customers. Additionally, they have a new application with a proven workflow to run via the CloudFlow Portal. This shall ease the implementation of other software products by using SES-Tec’s workflow as a tutorial. The bioreactor use-case uses 2D as well as 3D post-processing embedded in the CloudFlow Portal for a simplified analysis of the simulation results. Both post-processing technologies are developed by Fraunhofer IGD. They are reducing the downloading effort of the final results of the bioreactor simulation for the end user to a minimum. A Cloud-based infrastructure enables the development of more innovative and novel products:

□Strongly agree X Agree □Neutral □ Disagree □Strongly Disagree

A Cloud-based infrastructure enables more reliable and robust products.

□Strongly agree X Agree □Neutral □Disagree □Strongly Disagree Comments: Not the Cloud itself, but the availability of the simulation.

The integration of services on the Cloud within your development chain creates flexibility and

production on demand

□Strongly agree X Agree □Neutral □Disagree □Strongly Disagree

Services on the Cloud streamline and unclench the relevant processes

□Strongly agree X Agree □Neutral □Disagree □Strongly Disagree

5 BUSINESS MODEL (CO)

Business Model exploration concept The next figure summarizes the results of the working process employed with the software provider (AVL) involved in the Bioreactor Experiment in order to identify the cloud-based specific business concepts. For that, the different blocks of the Osterwalder methodology for business models generation were reviewed step by step, obtaining the following main concepts latter on tested during the Customer Development stage.


21

FIGURE 23: MODEL DEVELOPMENT STRATEGY

The full Business Model report generated can be found in Annex 4. Customer Development Following the Customer Development methodology and with the purpose of assessing the previous cloud-based business exploration concepts, the software provider identified one testing client that provided real feedback about such proposed business concepts through the answer to a specifically designed questionnaire. The client selected was SES-Tec and the results obtained can be found in Annex 5. Market prospection and exploitation expectative The next tables exposes market prospection data concerning aspects such as customer segments, market size or clients, sales and incomes expectations for the cloud-based exploitation intentions of AVL.

Specific know-how companies for non-

automotive (SES-Tec)

Certain engineering applications partners

Platform/infrastructu

re providers

Value chain companies (SES-Tec)

Pay-per-use

Pre-paid

Pharmaceutical process engineering (no experience on

cloud)

Mass market

Less closer relationship in general

Specific customization needs filtered/covered

by SES-Tec

Marketing strategy (“gaining of

momentum”)

Security

Quality of service

Upgrade of software

Service and maintenance of the

software

Lower price

HPC resources

Higher quality and lower cost for

product development

Infrastructure (HPC)

Cloud methods/interfaces

CloudFlow platform

Own technical capabilities

INDICATOR 1 year perspective 3 years perspect.

Number of proprietary

applications/workflows in the cloud

Quantify the number of applications or

worflows with other solutions to be exploited

in a cloud based manner

2 3-5

Customer segment/niche

(see slide 4 for details)

Define the type of customers to be

addressed in terms of sector,/industry,

customer profile, customer size (SME, etc.)

Pharmaceutical

industry, SMEs

Automotive,

general engineering; SMEs, LEs

Market size Quantify approximately the global market

size for that segment in terms of number of

buyers potentially demanding the product/

service

200 1500

Number of clients Quantify approximately the number of final

users that will pay for the product/service2 70-150

Market share Quantify approximately the percentage (in

terms of units or revenue) of the market

segment addresed that will buy the product

/service

1% 5-10%

Number of new jobs created Quantify approximately the number of jobs

created as a consequence of the cloud-

based model

1 2-5


22

FIGURE 24: MARKET PROSPECTION DATA

In terms of the charging models, in the specific case of AVL for the Bioreactor experiment the two-stage approach is:

1st stage: AVL has simplified the full approach and has basically followed the "on-demand" model. The cost of the application for the customer is 1 Euro core/hour.

2nd stage: AVL is interested in implementing also a pre-paid model for customers requiring simulations under a more intensive use (core unit price is cheaper).

As end user of the Bioreactor Experiment, SES-Tec presents the following exploitation figures.


Number of sales to existing clients Quantify approximately the number of unitary

sales of the cloud-based product /service to

already existing clients

1% 5-10%

Number of sales to new clients Quantify approximately the number of unitary

sales of the cloud-based product /service to

new clients

1% 10-15%

Average price

(see slide 4 for details)

Define approximately the average price or

prices of the cloud-services to the previous

clients

1 €/CPUh 0.8-0.9 €/CPUh

Total income Quantify the income derived from total sales 20.000 € 1.500.000 €

Production and commercial related

costs

Quantify approximately the total costs raised

from any type of activity associated to the

cloud-based business model

10.000 € 10.000 €/a

Payback Estimate the period of time (normally

expressed in years) required to recoup the

funds expended in the investment* or to

reach the break-even point

2-4 years


Customer segment/niche Define the type of customers to be addressed in

terms of sector/industry, customer profile,

customer size (SME, etc.)

Pharma and biopharma

industries are using

bioreactors for the

production of

biopharmaceutical.

Bioreactor manufacturer

will be adressed, which

are almost SMEs and

equipment manufacturer.

Other industries will

be also addressed,

such as automotive,

general engineering.

Market size Quantify approximately the global market size

for that segment in terms of number of buyers

potentially demanding the product

4.7$ billion +15-18% per

year

Number of clients Quantify approximately the number of final users

that will pay for the product1 per year Up to 5 per year


Market share Quantify approximately the percentage (in

terms of units or revenue) of the market

segment addresed that will buy the product

10.000-20.000€ 20.000-

50.000€/a

Company growth Quantify approximately the number of new

jobs created1 2

Total income Quantify the income derived from total sales 15% of total

turnover

+15% of actual

turnover only due to Cloud computing


23

Cloud benefits From an average end user perspective, the following use case has been created in order to reflect the benefits obtained by such end user based on the new cloud software utilization. Calculation and comparison of costs SES-Tec is planning to offer their customers simulation services, where huge computational resources are required in a short time, e.g. DoE analysis. In order to make a realistic calculation and evaluation of cloud benefits, two different simulation scenarios will be considered. The first one is a simple simulation, which lasts between three to four hours. The second one is a more challenging simulation and takes about one week of computational time per calculation and up to 25 parallel simulation variants. 1. Simple simulation (duration between 3-4 hours) Assumptions: An end user has already a subscription to a CFD software. They need to perform an average of one demanding calculation a day (in a limited time). The required time to perform a CFD computation locally with four cores on their test case is 3 hours 40 minutes. In order to make a fair comparison we will compare the traditional versus the cloud model for three years since this is the period to amortize the hardware (with an average use of one simulation a day meaning 365 simulations a year). Costs without using pay-per-use for the AVL/SES-Tec application: It is assumed that the 365 computations a year multiplied with three years result in 1095 simulations using a local machine with four cores and running every computation one after the other. TABLE 1: COSTS WITHOUT USING PAY PER USE

Matter of expense Costs in Euro

Software licencing 22000€ annual licence * 3 years 66000 € Hardware cost amortized over three years 5000 € Total cost task could be completed in 4015 hours or 167,3 days

(64.84€ per computation) 71000 €

or 64.84€ per computation

Costs with pay-per-use for the AVL/SES-Tec application: In this scheme 10 cores per computation are used, instead of just 4, thanks to the availability of the HPC infrastructure. TABLE 2: COSTS WITH PAY-PER-USE

Matter of expense Costs in Euro

Software costs 1095 CFD analysis x 1.5 hours x 10 cores x 1€ /CPU hour

16425 €

Hardware HPC cost 1095 CFD analysis x 1.5 hours x 10 cores x 0.06€ /core hour

985.5 €

Total cost and the task could be completed in 1642.5 hours or 68.4 days (15,9€ per computation), assuming one computation is launched at a time

17410.5 € or 15.90€ per computation

If more than one computation is launched at the same time, which is definitely possible using the HPC infrastructure, the time required to complete the task would be significantly less.


24

As it can be seen from these numbers, the cloud solution brings two benefits: 1. Cost reduction 75.5 percent (traditional model 71000 versus 17410.5 cloud model) 2. Time reduction 59.1 percent

Estimation of break-even point: The number of computations in the period of three years shows that the traditional model is cheaper

than the cloud model; 6667 computations equates to 2222 computations a year or 43 computations

a week or 6 computations a day (total traditional model costs 106000€ [22000€ a year * 3 years and

40000€ in 8 equipments] and the same cost for the cloud model).

So, if we consider a customer demanding over six computations a day then the traditional licence

model is cheaper, while if the customer should launch fewer than six computations a day, then the

cloud model is cheaper for them.

The benefits for the customer can be seen in detail in Table 3 (3 years and 365 simulations a year -

1095 computations):

TABLE 3: ECONOMIC VALUE SIMPLE SIMULATION

Before CF CF Experiment Difference Economic value

Software costs CFD software 22000€ a year

1€ per core/hour

For 3 years use:

75,11%

66000-(1095*10*1.5*1) = 66000-16425 =

49575€ Time to complete the task

3 hours 40 minutes

1 hour 28 minutes

(10 cores) 60% N/A

Expertise High High - -

Hardware costs Desktop

(5000€ in 3 years) HPC: 0.06€ core/hour

- 5000-1095*10*1.5*0.06=

5000-985.5= 4041.5€ TOTAL 53616.5€ in 3 years

2. More complex and time consuming simulation (DoE analysis): For a DoE analysis, up to 25 simulation variants have to be started in parallel. The required hardware and software resources are: 25 X 12 CPUs, 25 X AVL FIRE© base licence and 25 X 12 MPI (parallel) licences. Taking into account only one simulation variant, the hardware and software investment costs – in the case of an in-house calculation – can be summarized as follows:

1. Hardware: 1 X HPC (12-16 CPUs) = 10.000€ 2. Software: 1 X base and 16MPIs licences = 35.000€/a

Considering that one simulation variant takes about one week of computational time (7*24=168 hours) using between 12 and 16 CPUs, utilization of 80% per year, and cloud costs of 1.2€/CPUh the following comparison between cloud and in-house costs can be evaluated:


25

FIGURE 25: IN-HOUSE VS. CLOUD COSTS

Figure 25 compares computational costs (including hardware and licence costs) in the two different scenarios, cloud computing and in-house computation. The break-even point is reached with about 42 variants per year. This means that if our demand is less than 42 variants per year, cloud computing is still the cheaper solution than an in-house calculation. The real benefit of cloud computing can be seen in a case where in a short time several parallel variants have to be calculated/executed, such as the DoE analysis previously discussed with up to 25 simulation variants. The variants should be executed in parallel and the hardware and software investment costs would be 25 X 45.000€. In this case, the break-even point is reached with 1050 simulation variants per year. Currently, SMEs do not have these huge computational demands, so in-house investment for the HPC infrastructure is still not profitable and will clearly not be profitable for another period of 10 years. Without the cloud computing platforms, SES-Tec and similar simulation service providers are not able to offer simulation services where these huge computational resources are required.

0

20000

40000

60000

80000

100000

120000

0 20 40 60 80 100

Co

sts

[€]

Nr. of Variants

Cloud costs

In house costs


26

6 EXECUTION OF THE EXPERIMENT (CO)

For a complete description of the workflow on the cloud see Chapter 2 “DESCRIPTION OF THE

(ENGINEERING AND MANUFACTURING) PROCESS BASED ON CLOUD SERVICES”. The main activities

are summarized in the following list:

Collaboration with the CloudFlow Competence Centre CFCC

The user requirements were analysed and used for the design of the technical integration. A

new business model has been developed.

Introduction/integration of the software into the HPC/CloudFlow infrastructure

The software installation of AVL-FIRE® and the integration into the HPC/CloudFlow

infrastructure was performed. A test case was executed in the HPC infrastructure. Some bash

scripts have been written to ensure the data management after the simulation of the test

case.

Simple case investigation and cloud-based simulation

A simple simulation was started on the local machine and on the cloud. Both workflows are

compared and evaluated. Difficulties and potential errors were discussed with AVL, ARCTUR

and the CloudFlow Competence Centre.

Running of a real use-case on the cloud

A real-use simulation of a bioreactor was executed in the HPC infrastructure. Two simulations

in parallel were performed using 12 CPUs each. The bioreactor simulation results were

validated with academic reference literature. All scripts for the integration in the CloudFlow

Portal are finalised and the design of experiments DoE was prepared. Automatic evaluation

was partly achieved. The main evaluation of the DoE results was done on a local machine.

Workflow optimization and second simulation loop

All use-case simulation setup information is uploaded into the cloud. The workflow is

optimized and 15 bioreactor simulation variants have been conducted. The DoE analysis was

performed on the local machine after downloading the results.

Data evaluation and report writing

A summary of the performed work has been written.

Experiment assessment and validation

From a technical point of view there were two important progress steps conducted during the

project time within the CloudFlow platform using GW, namely the checkpointing and the

storage folder. Both folders were used for the final implementation in order to use 2D and 3D

post-processing. This leads to a better usability and a web-based visualisation of the

simulation results for the user.

Project management

The project organisation was conducted during the full project time of 12 months.

The implementation of the bioreactor use-case yields a working tutorial for how software can be

implemented into the CloudFlow Portal. This eases the integration of further software tools exploiting

cloud simulation services. Mainly, bash scripting is used for simulation execution and for data


27

management. It shows how to summarize data in an efficient way, which can be plotted in the 2D and

3D post-processing step. After the simulation, the DoE was performed on the local machine using

software tools for data analysis.

There were no deviations from the planned activities, except Activity 2, subtask “Integration of

simulation monitoring and stop function”. The option “stop of simulation” via GW is not finished at

the current time (January 2016). It is expected that this option will be ready to use by the end of April

2016.

In total, about 61.000 CPUh were used in the Cloud for the presented experiment.

Table 4 includes a description of the status and achieved results of each activity including their

subtasks.

TABLE 4: STATUS AND RESULTS OF EACH ACTIVITY

Activity No.

Description/achieved results Status

Activity 1 – Collaboration with CF CC:

Management Telco – It takes place every month (about two hours each). Different topics related to the organization and realization of the project are discussed

Finished

Technical Telco – Every two weeks, where technical topics related to integration in the cloud, are discussed.

Finished

User-requirements – after internal Kick-Off-Meeting in Graz (ARCTUR, SES-Tec, AVL), the user-requirements have been finalized and communicated to UNott (March 2015)

Finished

Business model – Telco with CARSA, AVL and SES-Tec regarding cloud specific business models for the experiment results (June 2015)

Finished

Business model – Telco with CARSA, AVL and SES-Tec regarding Cloud specific business models for the

experiment results (November-December 2015). Data evaluation and preparation of the reports regarding the business model by SES-Tec and AVL.

Finished

Activity 2 – Integration into HPC/Cloud:

Licences – Two separate licences files (for ARCTUR and SES-Tec) are provided by AVL short after internal Kick-Off-Meeting in Graz (March 2015)

Finished

Access to HPC – Access to HPC infrastructure by ARCTUR is provided in March 2015 incl. SSH tunnel to access the HPC and to run the simulations.

Finished

Software installation – Software installation by ARCTUR and SES-Tec in March 2015 Finished

Access via VNC – ARCTUR has provided access to HPC via a VNC viewer (March 2015) Finished

Access to CloudFlow Portal – Access is provided by SIMTEF and DFKI (June 2015) Finished

Virtual Machine VM – Access to VM provided by ARCTUR (June 2015). Finished

Workflow scheme – The workflow scheme has been developed and presented during a technical telco (Jun 2015).

Finished

Workflow integration into cloud – Analysis of the workflow integration is performed. Involved partners are SINTEF, DFKI, Fraunhofer, ARCTUR and SES-Tec.

Finished


28

VM – Installation of a virtual machine by ARCTUR, final initialization (July, 2015) Finished

Web-server – Installation of an apache web server by ARCTUR (July, 2015) Finished

GUI – Development of a web-based GUI (PHP and HTML) for the first evaluation in London (August, 2015) Finished

Run of simulation, monitoring, post-process in cloud – Integration and testing of the scripts into the cloud

(August, 2015) Finished

Discussion about SES-Tec workflow integration into CloudFlow Portal – Analysis of the workflow

integration has been done. Involved partners are SINTEF, DFKI, Fraunhofer, ARCTUR and SES-Tec (September 2015).

Finished

Integration into CloudFlow Portal – Run of first tests in CloudFlow Portal, test of GW integration in cloud,

which is provided by Fraunhofer, (October, 2015) Finished

3D post-processing in cloud – Preparation of scripts for the CGNS file writing. Testing of remote post-

processing tool developed by Fraunhofer. The required macros are written by AVL. Implementation in cloud is done by SES-Tec and Fraunhofer (November, 2015)

Finished

2D post-processing – Integration and testing of Fraunhofer 2D post processing tool in web-browser.

Preparation of required scripts, file formats, workflows, and test of these. Involved partners are SINTEF, DFKI, Fraunhofer, AVL and SES-Tec (December 2015).

Finished

Integration of simulation monitoring and stop functions – Basic functions are prepared by Fraunhofer.

Tests are carried out by SES-Tec. AVL has supported SES-Tec regarding to AVL-FIRE functionalities (December 2015 – January 2016)

Partially finished

Activity 3 – Test case:

Simple pipe flow simulation – SES-Tec has prepared a simple test case, which can be run very easily on the HPC. The test case is firstly tested by SES-Tec and after that by Fraunhofer. Run of the simulation on a node and cluster is tested (April 2015)

Finished

Simulation run via GridWorker –GridWorker was installed and tested locally by SES-Tec. Run of AVL-FIRE using GridWorker is firstly carried out locally by SES-Tec and after that by ARCTUR (May 2015)

Finished

Improvement of GridWorker runs – Improvement of GridWorker runs and development of running job scripts was mainly done by Fraunhofer. SES-Tec was responsible for the setup of the use-case (May 2015)

Finished

Activity 4 - Real case – bioreactor

CAD data preparation – SES-Tec has prepared a 3D model according to reference in DoW Finished

Simulation setup of the bioreactor – SES-Tec has generated the computational mesh and prepared the file containing simulation settings. After that, tests of the simulation are internally done and the results are compared with reference data.

Finished

Simulation run by ARCTUR – Run of the real case bioreactor simulation is successfully tested by ARCTUR. The tests are carried out by SES-Tec as well as Fraunhofer.

Finished

CGNS file – CGNS file is provided to Fraunhofer in order to test web-based post-processing (May 2015) Finished


29

Parameter study using GridWorker – SES-Tec has prepared the test case, where a parameter study can be carried out. Development of the GridWorker tools for the task is carried out by Fraunhofer.

Finished

Testing – Testing of the developed work flow and computational performances (SES-Tec and Fraunhofer). Finished

Running of a DoE analysis – A real case simulation is carried out during this activity by SES-Tec. Namely,

several process parameter are varied, such as aeration rate and impeller speed, in parallel, where each simulation is executed in parallel using GW.

Finished

Activity 5 –Workflow optimization:

Improvement of workflow: this activity includes optimization of the workflow. The scripts are re-written and optimized. All steps are discussed together with Fraunhofer and possible code/workflow modification are performed (November-December 2015). AVL has supported SES-Tec for the running scripts and macros optimization.

Finished

Activity 6 – Data evaluation/report:

Report writing: During this activity, simulation data are evaluated and prepared for the final report. Final

report writing finalized the activity (January-February 2016). Involved partners are SES-Tec and AVL. Finished

Activity 3 – Test case:

Simple pipe flow simulation – SES-Tec has prepared a simple test case, which can be run very easily on the HPC. The test case is firstly tested by SES-Tec and after that by Fraunhofer. Run of the simulation on a node and cluster is tested (April 2015)

Finished

Simulation run via GridWorker –GridWorker was installed and tested locally by SES-Tec. Run of AVL-FIRE using GridWorker is firstly carried out locally by SES-Tec and after that by ARCTUR (May 2015)

Finished

Improvement of GridWorker runs – Improvement of GridWorker runs and development of running job scripts was mainly done by Fraunhofer. SES-Tec was responsible for the setup of the use-case (May 2015)

Finished

Activity 7 – Experiment assessment and validation

Data evaluation: preparation of the test cases for the assessment and live demonstration, validation of the experiment, presentation via a phone conference, (December 2015)

Finished

Evaluation I: Preparation of the data and presentation for final assessment and demonstration in February 2016. TC in December 2015. Involved partners are SES-Tec and AVL.

Finished

Evaluation II: Final assessment and demonstration in February 2016 (January 2016) Finished


30

7 RECOMMENDATION TO THE CLOUDFLOW INFRASTRUCTURE (CO)

The CloudFlow Infrastructure was functioning well during the implementation of the bioreactor use-

case. Especially the new implementations within GridWorker are doing very well, namely the use of a

“storage” folder as well as the “checkpointing” folder. The “storage” folder is mainly used for storing

files without compression in order to be visualized in the 3D post-processing step. The “checkpointing”

folder is mainly used for transferring regularly information of a running simulation to the CloudFlow

Portal. Both folders are important for usability and control of the simulation service.

One minor technical improvement, which could help a lot to increase the usability of the CloudFlow

Portal, is a simple text editor. Sometimes there are little changes to make within the “configuration”

or “parameter” file. Now the workflow is as follows: the file is downloaded, it is changed and then it

is uploaded again. In order to simplify this workflow, it would be handy, if there is a text editor

embedded into the CloudFlow Portal, with which one can quickly do changes and save or overwrite

files. Maybe a new workflow can be implemented, which can be executed from the file browser via

right click.

Another suggestion for improvement is the implementation of the ability to copy a complete folder

within the file browser of the CloudFlow Portal. This could largely decrease the upload traffic, for

instance, when a working use-case can easily be copied and executed with different input parameters

instead of uploading the whole use-case to the SWIFT storage server again.

And last but not least working together with the CloudFlow team was very inspiring and made it

possible to intensely deal with cloud computing.


31

8 CONFIDENTIAL INFORMATION (CO)

There is no confidential information or data.


32

9 INVOLVED ORGANISATIONS

SES-Tec OG

SES-Tec is a spin-off of the Technical University of Graz and the K1 Competence Centre RCPE and was

founded in April 2013 in Graz, Austria. A close cooperation between SES-Tec and the University of Graz

and RCPE through several research projects is still present. Currently SES-Tec has four employees and

is specialized in the development of simulation methods and models for complex multi-physics

problems. SES-Tec customers are: Secop Austria GmbH, RCPE GmbH, BMW-Motoren GmbH, VENTREX

Automotive GmbH, Vogelbusch Biopharma GmbH, Polaris Industries etc. The core team of the

company consists of the two founders Dr. Dalibor Jajcevic and Dr. Wolfgang Lang. Both founders have

lots of experience in the field of virtual process development and application of numerical simulation

for complex multi-physics problems.

AVL LIST GmbH

AVL is the world's largest independent company for development, simulation and testing technology

of powertrains (hybrid, combustion engines, transmission, electric drive, batteries and software) for

passenger cars, trucks and large engines. AVL develops all kinds of powertrain systems as well as all

the instruments, systems and software required for powertrain and vehicle testing. In addition, AVL

develops and markets the simulation tools and methods which are necessary for the development

work. The developed simulation software is focusing on design and optimization of powertrain

systems and covers all powertrain components and phases of the development process.

In its business unit Advanced Simulation Technologies AVL is successfully developing and marketing

its general purpose 3D-CFD code AVL FIRE® since more than two decades. Besides the strong market

position in the automotive industry, AVL FIRE® is also increasingly used by academia, research

institutions and industry in various non-automotive application areas. These range from simulation of

single-phase and multi-phase flow processes in aeronautics and manufacturing industry to civil-

engineering, energy and environmental engineering, etc. With the participation in the proposed

project, AVL intends to strengthen its position as ISV in the non-automotive area by utilizing the

synergies in multi-phase, heat- and mass-transfer modelling in the automotive sector. With its

dedicated team of highly-skilled development and support engineers, AVL will contribute with

expertise in user-coding and high-performance computing incl. the know-how of porting user-coded

software elements onto different HPC hardware platforms and to achieve the required performance

optimization.

Arctur računalniški inženiring d.o.o.

ARCTUR is a leading service provider in the field of supercomputing in South Eastern Europe. HPC on-

demand, system administration services, code optimization and parallelization are offered through

the XaaS (Everything as a Service) model which provides our customers with substantial savings

compared to purchasing and maintaining their own equipment. Our advanced IT solutions (4PM, ADS)

have become indispensable tools for numerous companies and institutions.

High level of investment in research and development, exemplar cooperation with academic and

scientific institutions and excellent network of international partners enables us to pursue our

strategy, which continuous expansion into more complex computing environments and new global IT

markets.


33

10 EVALUATION DETAILS

Process overview

Instead of using AVL Fire, SES-tec used GridWorker (developed by Fraunhofer) to prepare the file for

the CloudFlow simulation. In GridWorker the end user defines the parameters of the simulation. The

file is compressed to be uploaded to CloudFlow.

In the CloudFlow Portal, the end user selects the files to be uploaded in the File Browser. This

process can take some time depending on the user’s internet speed. Then, the user right-clicks to go

to GridWorker. It is possible to check the progress of simulation through the ‘checkpoints’, which are

files created every few minutes. Residuals information is available so that the user can check that

everything is working as expected, which avoids wasting time or money if there is a problem.

The results are saved in the ‘simulation folder’, which contains the ‘checkpoint folder’ and the

‘storage folder’ with the results. In order to visualise the information, the post-processor developed

by Fraunhofer is available. Additionally, the end user can download the files to be opened in the

local machine with AVL software.


34

Issue 125.1: Cancelling a simulation in progress

Severity:

High

Description:

The end user is not able to cancel a simulation that has already started running.

Recommendations:

Integrate a stop function to the CloudFlow Portal so that the user can cancel an in-progress

simulation. At the time of final evaluation this issue had already been identified and UNott were

advised that the experiment partners are already working to resolve this.

Response from experiment leader:

Cancelation of complete workflows are possible through the Portal

Issue 125.2: Management of Resources from End -User Perspective

Severity:

High

Description:

At the time of the final evaluation, the end user had no way to know how much of the available

resources they have consumed. To get this information, it was necessary to call or e-mail Arctur. The

end user considers that it would be in everybody’s interest to have this information readily available.

Recommendations:

Implement an information window with the available resources from each end user, updated as

simulations are run.

Response from experiment leader:

This option will be included in the end version of CloudFlow Portal


35

APPENDIX 1: BASH SCRIPT (MAP.SH.GENERIC)

#!/bin/bash

PROJECT=BR1.fpr

CASE=Case

CPU=@cpus@

SOLVER=v2014

# Logging function

log ()

{

logTimestamp=`date –rfc-3339=seconds`

logComponent=”gridworker:mapper:script:”

logMessage=$1

echo “[$logTimestamp] $logComponent $logMessage” >> ${_LOGFILE}

}

# checkpointing function

check ()

{

# Old data is deleted

if [ -f “checkpoints/Case.fla” ]; then

/bin/rm –f checkpoints/Case.fla

fi

if [ -f “checkpoints/results.dat” ]; then

/bin/rm –f checkpoints/results.dat

fi

if [ -f “checkpoints/residuals.dat” ]; then

/bin/rm –f checkpoints/residuals.dat

fi

# Write the results.dat file

cp Calculation/${CASE}/Case.fla checkpoints/

fgrep “Formula: mean SHEAR-RATE” checkpoints/Case.fla | cut –f 9 –d “ “ > x1

fgrep “Formula: mean SHEAR-RATE” checkpoints/Case.fla | cut –f 11 –d “ “ > x2

fgrep “Formula: mean kLa” checkpoints/Case.fla | cut –f 11 –d “ “ > x3

echo “# time shear_rate kLa” > checkpoints/results.dat

echo “# [s] [1/s] [1/h]” >> checkpoints/results.dat

echo “# #000000 #FF4400 #0091FF” >> checkpoints/results.dat

paste x1 x2 x3 >> checkpoints/results.dat

# Write the residuals.dat file

grep –E “^ [[:digit:] ][[:digit:]] [[:digit:]]” checkpoints/Case.fla | cut –c 14-21 > resu

grep –E “^ [[:digit:] ][[:digit:]] [[:digit:]]” checkpoints/Case.fla | cut –c 23-30 > resv

grep –E “^ [[:digit:] ][[:digit:]] [[:digit:]]” checkpoints/Case.fla | cut –c 32-39 > resw

grep –E “^ [[:digit:] ][[:digit:]] [[:digit:]]” checkpoints/Case.fla | cut –c 41-48 > resm

# Write header of the residuals.dat file

echo “# Iteration ResU1 ResU2 ResV1 ResV2 ResW1 ResW2 ResM1” > checkpoints/residuals.dat

echo “# [1] [1] [1] [1] [1] [1] [1] [1]” >> checkpoints/residuals.dat

echo “# #000000 #FF4400 #ffb499 #0091FF #99d3ff #00FF08 #99ff9c #EEFF05” >>

checkpoints/residuals.dat

# Get the plot limit from resu

PLOTLIMIT=$(wc –l resu | cut –f 1 –d “ “)

PLOTLIMIT=$(( $PLOTLIMIT/2 ))

# Create a continuous number of Iterations

COUNTER=0

while [ $COUNTER –lt $PLOTLIMIT ]; do

INDEX[$COUNTER]=$(( $COUNTER+1 ))

let COUNTER=COUNTER+1

done

# Get every odd line/ get every even line

awk ‘NR % 2 == 0’ resu > resu1


awk ‘NR % 2 == 0’ resv > resv1


awk ‘NR % 2 == 0’ resw > resw1


awk ‘NR % 2 == 0’ resm > resm1

# Print the number of iteration

printf ‘%s\n’ “${INDEX[@]}” >> x4

# Write the residuals.dat

paste x4 resu1 resu2 resv1 resv2 resw1 resw2 resm1 >> checkpoints/residuals.dat

# Write the Case.fla

/bin/rm –f x4 resu1 resu2 resv1 resv2 resw1 resw2 resm resm1 checkpoints/Case.fla


36

log “checked”

}

# abort function

abort ()

{

cd Calculation/${CASE}

mv .isrunning .isstopping

log “aborted”

}

# simulate function

simulate ()

{

log “simulation started”

module load AVL_FIRE

# Create link in the checkpoints folder

if [ -d “checkpoints” ]; then

cd checkpoints

ln –s ../Calculation/${CASE}/${CASE}.fla checkpoint.log

cd ..

fi

# Run simulation

cd Calculation/${CASE}

fire_cmd \

-hosttype=x86_64-unknown-linux_r5i11 \

-solver_vers=${SOLVER} \

-project_dir=${_TASK_DIRECTORY} \

-mpi \

-cpu=${CPU} \

-project=${PROJECT} \

-case=${CASE} \

-noflb > run_fire.log

cd ${_TASK_DIRECTORY}

log “simulation completed”

}

# summarize function

summarize ()

{

# Write the results.dat file

cp Calculation/${CASE}/${CASE}.fla outputs/

fgrep “Formula: mean SHEAR-RATE” outputs/Case.fla | cut –f 9 –d “ “ > x1

fgrep “Formula: mean SHEAR-RATE” outputs/Case.fla | cut –f 11 –d “ “ > x2

fgrep “Formula: mean kLa” outputs/Case.fla | cut –f 11 –d “ “ > x3

# Write header of the results.dat file

echo “# time shear_rate kLa” > outputs/results.dat

echo “# [s] [1/s] [1/h]” >> outputs/results.dat

echo “# #000000 #FF4400 #0091FF” >> outputs/results.dat

paste x1 x2 x3 >> outputs/results.dat

/bin/rm –f x2 x3

# Write the residuals.dat file

grep –E “^ [[:digit:] ][[:digit:]] [[:digit:]]” outputs/Case.fla | cut –c 14-21 > resu

grep –E “^ [[:digit:] ][[:digit:]] [[:digit:]]” outputs/Case.fla | cut –c 23-30 > resv

grep –E “^ [[:digit:] ][[:digit:]] [[:digit:]]” outputs/Case.fla | cut –c 32-39 > resw

grep –E “^ [[:digit:] ][[:digit:]] [[:digit:]]” outputs/Case.fla | cut –c 41-48 > resm

# Write header of the residuals.dat file

echo “# Iteration ResU1 ResU2 ResV1 ResV2 ResW1 ResW2 ResM1” > outputs/residuals.dat

echo “# [1] [1] [1] [1] [1] [1] [1] [1]” >> outputs/residuals.dat

echo “# #000000 #FF4400 #ffb499 #0091FF #99d3ff #00FF08 #99ff9c #EEFF05” >> outputs/residuals.dat

# Get the plot limit from resu

PLOTLIMIT=$(wc –l resu | cut –f 1 –d “ “)

PLOTLIMIT=$(( $PLOTLIMIT/2 ))

# Create a continuous number of iterations

COUNTER=0

while [ $COUNTER –lt $PLOTLIMIT ]; do

INDEX[$COUNTER]=$(( $COUNTER+1 ))

let COUNTER=COUNTER+1

done

# Get every odd line/ get every even line




37





awk ‘NR % 2 == 0’ resm > resm1

# Print the number of iteration

printf ‘%s\n’ “${INDEX[@]}” >> x4

# Write the residuals.dat

paste x4 resu1 resu2 resv1 resv2 resw1 resw2 resm1 >> outputs/residuals.dat

# Remove auxiliary files

/bin/rm –f x4 resu1 resu2 resv1 resv2 resw1 resw2 resm resm1

# Write CGNS macro file

FINAL_ITERATION=$(tail -1 x1)

/bin/rm –f x1

part1=”PostConvertFileData( \”fpr\”, \””

part2=${_TASK_DIRECTORY}

part3=”/Meshes/TOTALAutoMesh.flm\”, \””

part5=”/Calculation/Case/Case.fl3\”, \”cgns\”, \””

part7=”/BioreaktorVB.cgns\”, \””

part9=”/BioreaktorVB.cgns\”,\”TI_”

part11=”\”,\”Flow:RelativePressure[Pa],Formula:Shear[1/s],Formula:kLa[1/h],Phase.01:

Velocity.U[m/s],Phase.01:Velocity.V[m/s],Phase.01:Velocity.W[m/s],Phase.01:VolumeFraction[-]\”,\”\”,\”\”);”

echo $part1$part2$part3$part2$part5$part2$part7$part2$part9$FINAL_ITERATION$part11 > CGNSoutput.macro

# Write CGNS file

xvfb-run fire_wm_offscreen –v=${SOLVER} CGNSoutput.macro > CGNS_convert.log

# Copy files into storage

if [ -f BioreaktorVB.cgns ]; then

mkdir storage

mv BioreaktorVB.cgns storage/

cp outputs/results.dat storage/

cp outputs/residuals.dat storage/

fi

log “results summarized”

}

# mapping function | STANDARD ROUTINE |

map ()

{

simulate

summarize

}

${_COMMAND}

exit 0


38

APPENDIX 2: USER REQUIREMENTS AND HOW THEY ARE MET

Partner Requirement Success Criteria Measuring Method Feasibility Success criteria achieved?

End User: SES-Tec

Access to HPC Access any time 24/7 (min. 95% of successful connection)

Test and evidence during the evaluation phase

High Yes – demonstrated in final evaluation. Connection now over 95%.

CPUs should be available

80% of required CPUs should always be available


High Yes – over 80%.

No idle time (no waiting time to run of a job)

Max idle time of ½ hour Test and evidence during execution of the experiment in the evaluation phase

High Demonstrated in final evaluation – just a few minutes idle time for running the job.

Fast data transfer Download: min. 2MB/s Upload: min. 0.5MB/s


High This is dependent on user connection – SES-Tec connection speed is not ideal at this time. But from the CloudFlow end this is achieved.

Simulation scalability with an increasing number of CPUs

Double number of CPUs leads to reduction of calculation time by minimum 30%

Test and evidence during the evaluation phase; will involve running simulation on a set number of CPUs, measuring calculation time, then increasing the number of CPUs and re-measuring the calculation time.

Medium Yes – up to 12 CPUs there is a reduction in calculation time, good scalability. Over 12 CPUs it is constant.

Reduction of calculation time

Reduction of calculation time (over in-house calculation) by 30% per variant

Measure during evaluation phase; comparison between Cloud-based HPC and in-house calculation

High SES-Tec has new in-house hardware which already gives good calculation time, so the CF solution does not show a reduction for 1 variant. But in-house it is not possible to do more than 5 in parallel, so for over 5 the difference is dramatic. For other users without in-house HPC the reduction in time for each variant would be substantial.


39

Economy (license & HPC costs)

No more than current in house CPUs costs, preferred -20%

Comparison of CloudFlow business model with in-house CPU costs during evaluation phase

High Yes. For more than 40 variants per year the licence cost would be a better option.

Simulation stability Maximum of 25% rerun simulations

Test and evidence during the evaluation phase. A minimum of 15 parallel different simulations will be run to determine rerun rate.

Medium Yes, the calculations are stable.

Data is produced for DoE (Design of Experiment) analysis

Run of up to 15 simulations To obtain number of finished simulations runs

Medium Yes, now up to 15.

Software Vendor: AVL List GmbH

Software installation is appropriate

Fast and easy software installation

Installation of new software versions in less than 3 hours

High The AVL software licence is available on demand. At the moment a report of hours from Arctur to SES-Tec is needed.

Usability of the Cloud

Easy to Use Usability Study conducted during evaluation phase

High From the end user perspective there is just one remaining issue with the stop functionality, but this should be resolved in February 2016. See usability evaluation notes in Appendix 2.

User defined post-processing is provided

Cloud simulation provides post-processing functionality

Successful demonstration of post-processing during evaluation phase

High Yes – 2D charts and 3D sections. This requirement is achieved with the integration of the post-processor developed by Fraunhofer during wave 1 and 2.

Job monitoring Availability of user interface for job monitoring

Integration into CloudFlow interface; check availability during evaluation phase

High Yes, check points integrated.

Licensing, controlling of used CPUS, evaluation of

Report for ISV containing CPUs hours, user names (user XY, XX CPUs hours)

Successful demonstration of functionality during evaluation phase

High Will be solved in next version of CF Portal.


40

CPUS hours are available for ISV

HPC Provider: Arctur d.o.o

Provide experiment with operational Cloud/HPC environment

Operational Cloud/HPC environment with respect to experiment specification

Comparison between experiment specification and established Cloud/HPC environment

High Yes – demonstrated in final evaluation.

Reduce time-to-solution for end user

Reduced time-to-solution compared to base case

Comparison of simulation time of final solution compared to base case

High Yes – for 25 variants there is a reduction from 5 weeks to ~1 week; more variants gives greater reduction.

Increase scalability of solution

Simulation runs scaled across multiple HPC nodes or Virtual Machines

Count of nodes or VMs used during evaluation phase

High Up to 12 CPUs is reduction of time, over 12 is no change – this is a general simulation issue, not specifically related to CloudFlow.

Reduced costs of simulation

Total cost of simulation is reduced

Calculate and compare total cost of simulation between base case and final solution

medium Below 40 variants cloud is cheaper; over 40 is break-even. Costs start to become very high with multiple licenses. It would be better if there were better value options when multiple licenses are purchased.


41

APPENDIX 3: USABILITY EVALUATION

The following methods were used for the usability evaluation:

1) heuristic evaluation in which two usability experts observed the end user while

performing a set of tasks on each application;

2) talk aloud in which the end user described their process as they were using the

application;

3) an interview of the end user following the software demonstration to explore the issues.

In this report, severities of the usability issues are identified and recommendations to resolve them

are proposed. It is recognised that it may not be possible to resolve each issue, and the suggestions

are for guidance only. High severity items should be addressed as a priority.

Summary of usability evaluation Pros:

End user was able to successfully use the software without technical difficulties.

Simulation progress can be monitored through checkpoints. Cons:

The user cannot cancel a simulation once running.

The user does not have sufficient information about resource management.

APPENDIX 4: BUSINESS MODEL EXPLORATION CONCEPT

Document Objective

This document reflects the results of the working process employed with the software provider (AVL)

involved in the Bioreactor Experiment in order to identify the cloud-based specific business concepts

to be tested during the Customer Development stage. With the purpose of assessing the cloud-based

D125.1 CloudFlow (FP7-2013-NMP-ICT-FoF- 609100)

42

business hypothesis presented in this document, the software provider will identify one or two testing

clients which will provide real feedback about the proposed concepts.

NOTE: in the document AVL will be referred as “the software provider”.

Value proposition

Some main concepts are to be stressed with regard to the value proposition offered to the customers

with the CFD software service in a cloud-based manner:

Very high computing power

Lower price specially for small companies (pay-per-use)

Product development with higher quality and lower costs

Moreover, in the cloud the application will help companies not using CFD technology and calculations

avoiding them to face huge investments (infrastructure, skilled engineers, etc.)in case this kind of

service could be required.

Distribution

Since the software provider will address a new market (pharmaceutical) there are non existing

customers and hence current channels to focus on.

For the cloud model it is foreseen that platform/infrastructure providers play a relevant role as a

distribution channel because of the visibility they have on the software provider’s application.

Companies belonging to the engineering value chain like SES-Tec, as participant in the CloudFlow

experiment, will also be considered as important in terms of the software service distribution.

Customer relationship

Having a new target in the form of a mass sales market when moving into the cloud, the kind of

relationship maintained with the customers will be less closer than it is currently in the automotive

sector (main focus industry at the moment), more oriented to solving customer requirements and

customization. For the new model in the cloud only some few customers will probably demand specific

adaptation works, that can be covered by partner companies such as SES-Tec. Anyhow, for the

pharmaceutical sector the specific customers requirements are expected to be firstly filtered by SES-

Tec.

Customers

In a cloud-based environment the customer segment targeted is completely new and not active up to

now. The intention is to extend the software service to the pharmaceutical process engineering area

as a market segment unknown for the software provider and assumed as a sector not having previous

work experience through the cloud.

Key resources

The main aspects to be stressed here are the ones related to the infrastructure provider (HPC) not only

for hardware but also for the management of the monitoring/accounting services, and the

development of methods and interfaces for the cloud where SES-Tec will take control of the

implementation tasks. Here a kind of platform like CloudFlow is considered as a key resource.

Apart from that the software provider will need to count on its own technical capabilities in terms of

both personnel and computer hardware resources.

Key activities

In the Cloud several activities in different areas (distribution, production...) are identified as key:


43

The marketing effort put on the solution offered is very important on the side of the

infrastructure provider, the software provider (AVL) and SES-Tec. The “gaining of momentum”

aspect should be intensified.

The security issue has to be 100% assured and therefore even more emphasized.

The quality of service needs to reach a very high standard.

The upgrade and availability of new software releases.

Key alliances

Besides keeping some existing partnerships with providers already operating in the cloud which

attends to the open collaboration approach of the software provider, some other alliance options are

also considered. In this sense, it will be relevant the partnership with companies like SES-Tec providing

specific know-how in non-automotive markets and sectors such as the pharmaceutical one as well as

with some other third partners when focusing on certain engineering applications.

Cost structure

The structure of costs will not suffer from relevant modifications when moving to the cloud specially

concerning what is a normal practice like additional costs associated to HPC or infrastructure oriented

providers. The revenue share will basically keep invariable in the new business model.

However, probably some extra costs related to service and maintenance of the software will have to

be faced after the new cloud-based model has been running for some time.

Revenue streams

The pay-per-use option is the one picked since it is more appropriate for the cloud-based

functionalities, in which the customer pays for the amount of CPU used for the needed calculations.

On the other hand, a pre-paid option would be also offered to customers requiring simulations under

a more intensive use, normally every day, but always restricted to a minimum established use. Here

CPU unit price is cheaper due to the higher volume of use.

Cloud-based business model exploration concept

We have reviewed step by step the main business blocks of the Osterwalder methodology for business

models generation. As a summary these are the main concepts to be tested during the Customer

Development stage.


44

APPENDIX 5: CUSTOMER DEVELOPMENT QUESTIONNAIRE

Segment

1. Do you use software solutions to support your engineering processes? 1. Yes, definitely

2. Yes, in some specific aspects

3. Not at all

Additional comments: ........................................................................................................................

2. What solutions do you use?

1. CAD, CAM, CAE

2. Simulation

3. PLM

4. Other


3. Do you use the software from AVL?

1. Yes, definitely

2. No, we use another similar software package

3. No, but we are interested in using it


Problems

4. Which are the main problems you have on CFD that you try to solve with the use of the software from AVL? Problems: General fluid flows in thermodynamics simulation (engine, compressors, bioreactors, tunnels.....).

5. Which of your main problems are not being addressed or cannot be addressed by the

software from AVL? Problems: Almost every problem can be solved with AVL software.

6. How would it be the best product to help you doing this job?

Details: AVL FIRE.

Current product

7. In how many of your engineering processes do you use the software from AVL? Details: About 60-70% of our processes. Others are faced with small computation on our own.

8. How often do you use the software from AVL?

Details: Every day.


45

9. How many people in your organization use the software from AVL? Details: 2 persons.

10. How many licenses of the software do you currently have/need? Details: 5 single licenses and 32 MPI (Message Passing Interface) in parallel.

11. Besides licensing, is there any other charging method offered to you by AVL? Details: No.

12. How much do you approximately pay a year for the software from AVL? Details: About 40.000 € for all licenses.

13. Under the AVL’s software product/service that you currently pay for, with which of the following business aspects you are not sufficiently satisfied or you consider that might be improved, if any?

1. The distribution channels and means utilized to provide the service

2. The type of relationship you keep with the software provider

3. The price and/or way of charging applied to the software product/service

4. Technical aspects (accessibility, reliability, usability, flexibility, etc.) that other providers’

solutions improve

Additional comments: None.

14. Generally speaking, which are the strong points of the software from AVL? Points: Availability of technology for IC-engine simulation and related application in the automotive industry.

15. Generally speaking, which are the weak points of the software from AVL?

Points: Some applications need workaround (not straightforward).

Cloud-Computing

16. Do you use the Cloud computing services from any provider?

1. Yes, we do

2. No, we are not interested in using Cloud services

3. No, but we are interested in using them


17. What is your opinion about Cloud computing?

1. Positive

2. Negative

Why positive/negative: Usage on demand, HPC, lower costs,.....

18. Do you consider Security to be a constraint for a software service in case of migration to a cloud based environment?

1. Not at all


46

2. We are not aware of potential problems concerning security when operating in the

cloud

3. Yes, but not objection is to be put as long as security is somehow assured in the

service

4. Yes, but only under some specific conditions

5. Yes, it is considered as a constraint hardly solved


Cloud product

19. Do you consider AVL’s software service might raise advantages (price, reliability, computing power, relationship with the provider, after-sales service, etc.) to your organization in case of being offered in a cloud-based manner? 1. Yes, definitely

2. Yes, in some specific aspects

3. Yes, but not relevant advantages are to be expected

4. We are not aware of potential advantages coming from cloud-based software services

5. Not at all


20. Which are the main technical/functional problems or weak points that might be solved in case of accessing to the AVL’s software solution in a cloud based manner? Please specify your answer: The pre-processing phase (followed by solving and then post-processing phases) consumes a lot of resources and today we prepare the model in-house. In the cloud we could do it online and do not need to have an in-house license.

21. With numerical marks, being 1 the highest and 8 the lowest, specify what you expect to

be the (positive) impact in each of the following aspects when moving to a cloud based software service

1. Lower price - 1

2. Service flexibility - 7

3. Service customization - 6

4. Service availability - 9

5. Higher computing power - 2

6. Service accessibility - 3

7. Service usability - 4

8. Service performance - 8

9. Service reliability - 5

Add any other aspects if necessary: Numbers indicate priority from most to less important.

22. Are there aspects concerning the channels/means the AVL’s software solution is

provided that might be improved in a cloud based manner? Please specify your answer: ......................................................................................................................................................

23. Through which of the following channels would you prefer to be served and maintain post-sales communication and relations with AVL as software service provider in the cloud?

1. Physical sales force

2. Web/online sales force


47

3. Cloud infrastructure/platform provider

4. Third companies in the engineering/manufacturing value chain as intermediaries

Add any other channels if necessary:.....................................................................................

24. Are there aspects adding value to the AVL’s software service received in case the relationship with the provider might be enhanced or modified according to the characteristics of a cloud based environment? Please specify your answer: ......................................................................................................................................................

25. Which types of relationship would you value most to establish and maintain with AVL as software service provider in the cloud?

1. Co-creation (R&D, etc.)

2. Particular needs solving

3. Short-term base

4. Long-term base

5. After-sales/maintenance/upgrade

6. Training/consultancy

Add any other type of relationship if necessary:.................................................................................

26. Besides the AVL’s software functionalities themselves are there other service related activities or aspects that would need to be enhanced in a cloud based manner? Please specify your answer: ......................................................................................................................................................

27. With numerical marks, being 1 the highest and 9 the lowest, specify how you value

each of the following activities/aspects when moving to a cloud based software service

1. After-sales service and support - 8

2. Software upgrade - 7

3. Quality of service - 6

4. Security assurance - 5

5. Training support - 9

6. Consultancy oriented service - 1

7. Offering of new cloud computing services and possibilities - 2

8. Provider proximity and knowledge of our needs and expectations - 3

9. Joint R&D activity and collaboration - 4

Add any other activities/aspects if necessary: Numbers indicate priority from most to less important.

Price and payment model

28. What is your opinion about the unitary price that AVL is proposing to charge for the software service offered from the cloud? 1. It is fair attending to the value provided

2. It is high according to what I had expected

3. It is low attending to the value provided

Additionaln comments:......................................................................................................


48

29. What is your opinion about the charging method that AVL is proposing to implement for the software service offered from the cloud?

1. It is adequate and responds to the characteristics of a cloud based software service

2. It is worthy for a cloud model but other charging alternatives would also be desired

3. It does not raise any added-value and does not take advantage of a cloud based

service possibilities

Additional comments:........................................................................................................

30. With numerical marks, being 1 the highest and 3 the lowest, specify how you value each of the following charging methods when moving to a cloud based software service

1. Pay-per-use - 1

2. Flat rate - 2

3. License subscription - 3

4. Other

Add any other charging method if necessary:.......................................................................

Software services workflows offered by the “CloudFlow” platform

31. In the context of the usage that your organization makes of engineering/manufacturing oriented software, would you demand different software services/functionalities provided separately from different software products to be connected or integrated in a single software solution?

1. No, never

2. Yes, but punctually for some very specific applications

3. Yes, increasingly

4. Yes, definitely

Additional comments: ....................................................................................................

32. Do you consider that having access to a platform offering such a type of integral solution and connected services and workflows your organization may incur in resources, efforts, time and money saving?

1. Not really

2. Yes, but only for some very specific applications


4. Yes, definitely

Additional comments:....................................................................................................

33. Would you be willing to pay an increased and fair unitary price in order to make use of such a type of integral software service in the cloud allowing workflows among different software solutions accessed through a single platform?

1. Not at all

2. We are not able to valorise such a type of service

3. Yes, but punctually for some very specific applications

4. Yes, but under certain conditions (please specify)

5. Yes, but firstly testing the solution


7. Yes, definitely

Additional comments:.........................................................................................................................

Documents

EXPERIMENT DESCRIPTION AND EVALUATION - CORDIS · 2017-05-11 · computation. AVL-List as independent software vendor in the experiment benefits by new customers or more sold licences