ADVANCES IN CFD MODELLING FOR DATA CENTRE OPTIMISATION

AIRAH and IBPSA’s Australasian Building Simulation 2017 Conference, Melbourne, November 15-16.

1

ADVANCES IN CFD MODELLING

FOR DATA CENTRE OPTIMISATION

NEIL SILKE, PhD, BE Lead Engineer

Data Centre CFD

WARWICK STANNUS, M. AIRAH, MBA, BE(HONS) Group Engineering Manager

A.G. Coombs

ABSTRACT

CFD modelling has become established practice for assessing and improving data centre

performance both in design and in operation. However, with increasing rack power densities and

new cooling methods coming to market, CFD is now an essential component of the design

development process and is considered by many data centre owners as a pre-requisite for

demonstrating design compliance for both normal operating conditions and various failure mode

scenarios.

This increasing reliance on CFD models by designers and clients requires more stringent modelling

of the data centre built environment, process air systems, racks and equipment fit-off. This paper

will detail a number of modelling techniques developed to respond to these increasing demands.

The modelling techniques will be illustrated using a recent project that involved high density racks

with stringent requirements relating to system resilience and rack inlet temperatures as a case study.

The project required the development of detailed CFD models leveraging the Revit architectural

and services models as well as leading edge equipment rack models. Ultimately optimisation of the

design impacted the data centre layout, rack build specifications and thermal plant design.

INTRODUCTION

Data centre design has undergone significant transformation in the past decade as a result of the

introduction of new design standards such as ASHRAE's “Thermal Guidelines for Data

Processing Environments” and TIA 942-A Telecommunications Infrastructure Standard for

Data Centres. These standards have supported the development of a range of new process

cooling technologies and designs that see today’s data centres achieving significantly higher

levels of energy efficiency without compromising service delivery.

This paper reviews the use of CFD to support the design, construction, commissioning and life

cycle management and operation of a data centre that will accommodate some 80 ICT cabinets

and an ICT equipment load of 350kW. Whilst not large by data centre standards, the data centre

is of national importance and as such the client brief required the design to demonstrate

compliance with the following targets:

• Availability 0.999997

• Mean Time Between Outage (MTDO) 200,000 hours

• Mean Time to Repair 40 minutes


2

In addition, the data centre was required to meet the design, construction and operation

requirements for an ANSI/ TIA-942 – Tier 3 compliant data centre.

Supporting these design parameters were more than 20 client technical design standards.

1. DESIGN MANAGEMENT

Design management is recognised as a key success factor in the delivery of any project and this

is particularly the case for data centres that involve complex design requirements and short

delivery programs.

A number of design management decisions were made to support the effective use of CFD and

statistical modelling on the project. These decisions recognised the need to resolve a range of

complex inter-related design issues and to prove design compliance against a stringent design

approval and risk management assessment process.

1.1 Early engagement of modelling specialists

From the outset of the project the challenges involved with validating the design were

recognised and two key sub-consultants were engaged early to undertake the required

availability and reliability studies and CFD modelling. This early engagement strategy allowed

the modellers to contribute their experience and expertise to the project early in the design

process, thereby maximising the value of their contributions. It was also effective in mitigating

the design program risk as the majority of issues were able to be avoided rather than discovered

during the design validation modelling.

1.2 Development of a reverse design brief

With such a large number of technical standards as well as a number of updates to the client’s

requirements through the pre-design phase it was important to distil a clear statement of client

requirements prior to the commencement of design development. This was achieved through the

issue of a “reverse brief” that provided the project design team, including our specialist sub-

consultants with structure and clarity to the project objectives and requirements. Importantly, it

also provided a framework against which all subsequent design decisions were assessed.

1.3 Sub-consultant briefing

The requirements to validate the design required both CFD modelling of normal operation and

various failure modes. Whilst the modelling requirements for steady state normal modes of

operation are relatively straightforward to determine, analysis of critical failure modes can be

more complex.

The first step in the process is to build a clear understanding of the reliability, availability and

maintainability factors of the system architecture, and in this case to remove any single points of

failure. This process relies on the development of the system air, water and power schematics,

as well as the functional control strategies. This, however, is not enough, and a detailed

understanding of how equipment and also systems respond to external events such as a loss of

power needs to be understood.


3

The result of this initial study informed the failure modes that were subsequently requested to be

modelled using CFD.

1.4 Use of BIM

The architectural, structural and services designers committed to the use of BIM modelling from

the start of the design, and the models were progressively developed through the project

delivery to provide the client with as-built FM-ready models.

The benefit of this integrated model workflow was that it supported a wide range of design and

construction activities and benefited from the use of both constructible services modelling using

the BIM-MEPAUS Revit Template Add-in as well as detailed architectural and structural content

such as the raised-floor system. This meant also that many aspects of the design and virtual

build effectively happened concurrently, eliminating the risk of design intent being lost through

a disconnected construction-documentation process.

2. USE OF CFD ON THE PROJECT

Computational Fluid Dynamics (CFD) utilises computer-based numerical analysis for systems

which involve fluids such as air. The simulations outlined in this paper were performed using the

CFD code called 6SigmaDCX which was provided by Future Facilities Ltd. This paper includes

information regarding the extent of the modelling detail applied and associated modelling

techniques. It covers some examples of the results and their implications. It also addresses desirable

approaches to utilising CFD analysis at design, during integrated systems testing (IST) and then

continuously during the day-to-day operation of the data centre.

The paper focuses on some specific modelling advances, which are associated with the data-centre-

focused CFD software tool utilised. While some of these methodologies may be available in

general-purpose CFD codes they are not currently available, to our knowledge, in any other data-

centre-specific CFD software.

2.1 CFD Software and modelling philosophy

The CFD software 6SigmaDCX was used as it is specifically designed for analysing data centres

and their associated plant. The extensive vendor libraries and data-centre-focused reporting allowed

for efficient model creation and reporting. Due to the project’s early stages, many decisions were

still at the conceptual level, requiring suitable modelling assumptions. This also included the

recommended approach of modelling the worst-case scenario to stress test the capabilities of the

design.

2.2 Modelling Detail

A model, by definition, is less complex than the system, and so simplifications are required. It is

explained by Versteeg and Malalasekera [1] that “the results generated by a CFD code are at best as

good as the physics embedded in it and at worst as good as the operator”. A benefit of utilising a

CFD code specifically designed for modelling data centres is that it helps remove some of the

choices regarding modelling detail required by the operator, when compared with utilising general-

purpose CFD codes. The software contains a series of standard objects that are utilised in a data

centre (e.g. racks, cable routes, CRACs), and also a library of thousands of data centre items that


4

represent vendor items (e.g. an HP Proliant DL380 G9). Additionally, the gridding and solution

configuration are automatically adapted to the flow regimes expected within a data centre. This

allows for a relatively rapid model build.

All models represented steady-state conditions. The automated gridding calculated a desired grid

based on objects in the model and their airflow behaviour. An example of this is seen regarding the

IT equipment, where the automated gridding focused grid around the air inlets and outlets, and

especially in the direction air will travel to ensure accurate representation of the air movement.

The automatic grid generated was a 42 million (685x103x598) structured Cartesian grid, which was

our rough limit due to RAM. A further model with a reduced grid of 10 million cells (417x63x360)

gave good results agreement and was therefore used for continued model solving throughout the

rest of the project. The software utilises the KE turbulence model, and the treatment of turbulence is

adapted automatically near the wall, dependent upon the grid size.

The data centre was designed to be well sealed; however, certain areas of leakage were represented,

which included but was not limited to door gaps and internal rack leakages. Generally, if the

leakage gaps are relatively large they are defined using an explicit gap geometry, which will be

gridded. If the gaps are relatively small (compared to the grid) then a larger perforated obstruction

with a representative free area ration is used. Convergence was judged using a standard approach to

residual termination, where errors are summed and normalised for each grid cell. However, further

verification was assessed using sensors values, which are reported upon throughout the solving

progress. The software automatically created sensors in areas of interest, for example at the front

faces of the racks, while additional sensors can be applied by the user.

6SigmaDCX was first released in 2004, and since then has been utilised by world’s leading data

centre consultants and owner operators. For trouble-shooting exercises and existing data centre

analyses the predictions are usually compared with physical measurements. For design models,

verification can be incorporated during the IST stage. Our target is to obtain predictions within 10%

of measurement, and is usually achievable when appropriate data is used to build the model. The

software has been validated using benchmark theoretical cases, comparisons with measurements

from operational data centres, as well as independent validation from customers and research bodies

such as The National Science Foundation. An example of a validated data centre CFD model using

6SigmaDCX is provided by Ruiz [2].

This paper will not cover every detail and simplification included within the modelling process

adopted, but highlights some of the key modelling techniques. These key areas are the IT

equipment, CRAC (Computer Room Air Conditioner) and associated controls and rack

representations.

2.3 Rack representation

This project utilised a hot-aisle containment solution. The reported advantages of containment

solutions are extensive [3, 4], while in our experience the specifics required to ensure a fit-for-

purpose containment solution as less well reported. For example, it should be noted that a well-

designed containment solution may be compromised through the choice of a rack which does not

segregate the hot from the cold internally. As with most conceptual data centre design projects, the

choice of rack was undecided so suitable configuration assumptions had to be made. It was agreed


5

early on that an internally well-sealed rack must be utilised, but not to a point of being un-

realistically achievable.

Racks built on previous projects and library racks were inspected to ensure that the rack model

fitted the above requirements. The assumptions utilised were reported to the client as part of the

project documentation. Figure 1 shows the key areas assessed with regards the expected areas of

leakage. Representative gap dimensions were reported back to the client. All leakage gaps were less

than 5mm.

Figure 1. Areas of expected leakages for a rack are shown including the Side gaps between the

mounting rails and the cabinet sides, U-Slot blanking gaps and the Top and bottom mounting rail

gaps

2.4 IT equipment representation

When modelling a data centre, it is not practical to model the individual printed circuit boards and

electronic packages associated within each piece of IT equipment. Therefore, the IT is usually

represented as a “black box”. In this case the IT (and CRAC) black boxes utilise a surface-heat flux

to represent the heat source/sink. For this project, the IT equipment was unknown therefore generic

4U servers were utilised from the software’s design library. The generic servers take their air in

through the front and eject it out the rear.

In early releases of the software the IT black boxes utilised an explicit air-flow volume definition

for the servers. Therefore, the model would take exactly the airflow required, and no more or no

less. In reality, when containment is employed, there is the potential that the contained area will

become pressurised, and therefore affect the IT fan operation. The impact of pressure driven flows

through IT equipment is noted by Kennedy [5] who reports “a pressure difference of 10Pa may

increase the flow rate through the servers by 25%”.

The implications of excluding this effect could have a strong impact upon the results of a CFD

model depicting containment. The general aim is to provide a slight excess of air so that a good

thermal environment is provided for all the IT, without being wasteful.

This project therefore utilised a feature within the software to allow pressure-driven excess flow.

This feature was initially introduced in 2011 as a mainstream addition to the software. The

coefficients associated with this parameter are not available for publication, but are based upon


6

detailed CFD models of IT equipment. So, although the flowrate is defined using an assumption of

56.6 L/s of air per kW, this flow rate may vary dependent upon the pressure across the equipment.

2.5 Cooling modelling and associated controls

Recent design standards, technological advances, and the increased importance of running data

centres greener, have led designers and end-users to utilise more sophisticated means of ensuring

redundancy while not being inefficient in normal operating modes. Specifically, the use of

containment, EC fans, free cooling and associated control mechanisms are of increasing popularity.

It was important to include as much detail with regards the cooling unit representation and also to

ensure that the modelled control system was an accurate representation of the control methodology

used in reality. Figure 2 shows the impact different fan representations have upon the predicted air-

flow patterns.

Figure 2. Impact of different fans upon the resultant airflow patterns

The CRAC unit model included a variable cooling capacity dependent upon the return-air

temperature to represent the selections provided by the manufacturer. The CRACs had a maximum

flow rate of 5,000L/s and a net sensible cooling capacity of 97kW at 35°C return-air temperature.

The software allows for a choice of fan type and configurations including uniform flow, blowers or

radial fans. The CRAC units had lowered EC plug fans, so the radial fan was chosen. The CRAC

support housing was also modelled. Figure 3 shows the CRAC unit.

EC PLUG FANS

SCROLL FAN


7

Figure 3. CRAC Unit with lowered EC plug fans

It was important that the software was able to replicate the controls mechanism realistically.

The CRACs were controlled individually to a supply-air temperature set point of 21°C. However,

the CRAC fans were group controlled to an averaged pressure difference of 15Pa across the raised

floor using multiple sensors. Figure 4 shows a diagrammatic representation of the pressure controls

as extracted from the CFD software. The CFD software implements the control analysis by utilising

a proportional-integral controller, which is embedded within the normal iterative solution.

Figure 4. CRAC Cooling and airflow controls diagram


8

2.6 Normal operation and CRAC failure mode

A great strength of modelling is the ability to run unlimited scenarios in a risk-free environment.

This includes investigating how the data centre will perform under failure scenarios. It was

important that the controls system could automatically adapt to a failure and ensure a resilient

thermal environment for the IT. Running experiments allowed for an optimised floor grille damper

position and associated pressure set-point that would achieve this. In the normal operating mode,

the fans were running at just over 50% (allowing for significant energy savings) and during the

failure mode were running at 100%.

2.7 Reporting – modelling allows understanding and communication

Each model was reported upon extensively. This section focuses on some key issues with the design

highlighted by the software and how the use of the model enabled possible solutions to be tried and

tested before recommendations were made to the client.

Figure 5 shows the maximum inlet temperature for IT within the racks while the data centre was in

failure mode. The failed CRACs are indicated with a red cross (X).

Figure 5. Maximum Inlet temperature for the IT equipment in Failure Mode

It was clear that some of the racks were seeing elevated inlet temperatures. Racks on the side near

the failed CRACs (Rack A) were experiencing hotter inlet air temperatures when compared with the

racks on the opposing side.

Figure 6 shows virtual streamlines coloured by temperature located at the IT inlets for Rack A and

Rack B. It was evident that some of the air entering the IT equipment in Rack A was sourced from

the hot rear side of the rack, as is clear from the red streamlines. This hot air was not evident for

Rack B.

Rack A Rack B


9

Figure 6. Virtual streamlines located at the IT inlets

Figure 7, which shows a pressure plane at 2m above the raised floor, gave some understanding as to

the cause of this effect. It is evident that the pressure along the hot aisle varies. The pressure in the

hot aisle near Rack B is less than the pressure in the cold aisle, therefore encouraging air to go from

the cold to the hot aisle (highlighted with a blue box ) . However the pressure in the hot aisle

near Rack A is greater than the pressure in the cold aisle, therefore encouraging the air to move

from the hot aisle to the cold aisle ( ). This effect occurred even with a relatively well-sealed

rack. While there were possible other remedies the client agreed to reduce the CRAC temperature

set-point which in turn brought the inlet temperatures down under the agreed SLAs.

Figure 7. Pressure plane at 2m above raised floor in Failure Mode

Rack A Rack B


10

2.8 Next Steps and the Future

With regards this specific project, future plans to utilise the 6SigmaDCX CFD tool beyond this

phase of the project have not been agreed. There is significant value in incorporating CFD

modelling as part of the IST and beyond into operational planning. At the IST phase, measured

performance is used to validate the model but also confirms that the commissioned data centre

matches the customer’s original design requirements, because it is quite common for aspects of the

design to vary during the build process.

Finally, we believe there is further benefit for adopting the use of CFD during the life of the data

centre. The Green Grid incorporates CFD modelling as a requirement for Levels 3 and 4 of their

Performance Indicator Data Centre Assessment Guide [6]. Ad-hoc CFD analysis can be used to

assess the full extent of a data centre’s inability to achieve their capacity requirements, yet proactive

on-going CFD use can help ensure that lost or stranded capacity is kept to a bare minimum. An

example of this is given [7].

3. DESIGN SPECIFICATIONS

Both the CFD and the Availability and Reliability studies and reports need to be translated into

a built performance. The translation of the findings and recommendations arising from the

studies can be difficult to implement and requires careful attention to detail, particularly in

relation to system functional control descriptions and equipment selections.

4. COMMISSIONING

Ultimately, system performance needs to be validated through the commissioning process.

Construction phase pre-commissioning inspection test plans can assure that the installation has

been built to the required standard, whilst the component commissioning places the equipment

into operation. However, it is the integrated system testing and site acceptance testing that are

key, and these reference the Availability and Reliability and CFD Reports in a similar way that

the fire mode cause-and-effect integrated test plan should reference the Fire Engineering Report.

The following lessons were learnt on the project through the commissioning process.

The use of large floor-mounted heater banks or fan heaters mounted in racks is generally

inadequate to validate system performance with any degree of confidence. Far greater

confidence can be gained through use of rack-mounted server emulators that are able to more

accurately represent servers through their ability vary heat output and airflow. Notwithstanding

these limitations, the full heat-load test can provide valuable insights to the systems dynamic

response under various failure modes.

Where cold isle containment is used, site acceptance testing using a single fully built-out

module combined with heater banks for the balance of the floor load may well provide a better

basis for acceptance testing.


11

5. OPERATION

The CFD modelling showed the importance of maintaining stringent standards with respect to

sealing of the server cabinets to avoid recirculation of hot air back into the cold air. It appears

that these requirements are rarely communicated to the data centre operator and provision of

point of use information and components such as blanking strips are important in maintaining

the long term efficient operation of the data centre.

Finally, there can be significant value in utilising CFD simulation techniques over the life of the

data centre. While some progressive companies are doing just that, it is expected that this approach

will become more popular over time.

CONCLUSION

The use of CFD to inform data centre design, commissioning and long-term operational

management provides significant opportunities to improve their performance in terms of energy

efficiency and resilience.

To assure the value of CFD modelling is maximised, the design management plan should see

early engagement of the modeller and the integration of the CFD modelling findings and

recommendations into the design and commissioning process.

With increasing power densities, the use of CFD modelling needs to be matched with more

stringent testing regimes that see the completed facility, including the racks and containment

systems, tested with server emulators rather than the common practice of more straightforward

floor-mounted fan-heater-based acceptance tests.


12

REFERENCES

(a) Books and handbooks

1. Versteeg, H. K. and Malalasekera, W. ; An Introduction to Computational Fluid Dynamics, pp.5-

6, 1995

6. The Green Grid ;The Performance Indicator Assessing and Visualizing Data Center Cooling

Performance.

(b) Websites

2. Ruiz, J. ; The Calibrated Data Center: Using Predictive

Modeling,https://journal.uptimeinstitute.com/the-calibrated-data-center-using-predictive-modeling/

3. http://www.datacenterknowledge.com/archives/2012/11/15/benefits-of-data-center-containment/

4. http://www.upsite.com/blog/evolution-data-center-containment/

5. Kennedy, D.; Ramification of Server Airflow Leakage in Data Centers with Aisle

Containment, 2012, https://www.tateinc.com/sites/default/files/support-docs/tate-

ramificationleakageaislecontainment.pdf

7. Bana, M. and Docca, A, and Davies, S. ; From Compromised to Optimized: One Data Center:

$10 million Saved, https://www.futurefacilities.com/media-centre/whitepapers/from-compromised-

to-optimised-one-data-center-10-million-saved/

http://www.upsite.com/blog/evolution-data-center-containment/

https://www.tateinc.com/sites/default/files/support-docs/tate-ramificationleakageaislecontainment.pdf

https://www.tateinc.com/sites/default/files/support-docs/tate-ramificationleakageaislecontainment.pdf

https://www.futurefacilities.com/media-centre/whitepapers/from-compromised-to-optimised-one-data-center-10-million-saved/

https://www.futurefacilities.com/media-centre/whitepapers/from-compromised-to-optimised-one-data-center-10-million-saved/

Documents

ADVANCES IN CFD MODELLING FOR DATA CENTRE OPTIMISATION