Reproducible and replicable CFD:its harder than you think
Completing a full replication study of our previously published findings on bluff-bodyaerodynamics was harder than we thought. Despite the fact that we have goodreproducible-research practices, sharing our code and data openly. Heres what welearned from three years, four CFD codes and hundreds of runs.
Olivier Mesnard, Lorena A. BarbaMechanical and Aerospace Engineering, George Washington University, Washington DC 20052
Our research group prides itself forhaving adopted Reproducible Re-search practices. Barba made a pub-lic pledge titled Reproducibility PIManifesto1 (PI: Principal Investigator), which atthe core is a promise to make all research mate-rials and methods open access and discoverable:releasing code, data and analysis/visualizationscripts.
In 2014, we published a study on Physics ofFluids titled Lift and wakes of flying snakes.2
It is a study that uses our in-house code for solv-ing the equations of fluid motion in two dimen-sions (2D), with a solution approach called theimmersed boundary method. The key of sucha method for solving the equations is that it ex-changes complexity in the mesh generation stepfor complexity in the application of boundaryconditions. It makes possible to use a simple dis-cretization mesh (structured Cartesian), but at thecost of an elaborate process that interpolates val-ues of fluid velocity at the boundary points to en-sure the no-slip boundary condition (that fluidsticks to a wall). The main finding of our studyon wakes of flying snakes was that the 2D sec-tion with anatomically correct geometry for thesnakes body experiences lift enhancement at agiven angle of attack. A previous experimentalstudy had already shown that the lift coefficientof a snake cross section in a wind tunnel gets anextra oomph of lift at 35 degrees angle-of-attack.Our simulations showed the same feature in theplot of lift coefficient.3 Many detailed observa-tions of the wake (visualized from the fluid-flowsolution in terms of the vorticity field in spaceand time) allowed us to give an explanation ofthe mechanism providing extra lift.
When a computational research group pro-duces this kind of study with an in-house code,it can take one, two or even three years to write afull research software from scratch, and completeverification and validation. Often, one gets thequestion: why not use a commercial CFD pack-age? (CFD: computational fluid dynamics.) Whynot use another research groups open-sourcecode? Doesnt it take much longer to write yetanother CFD solver than to use existing code?Beyond reasons that have to do with inventingnew methods, its a good question. To exploreusing an existing CFD solver for future research,we decided to first complete a full replication ofour previous results with these alternatives. Ourcommitment to open-source software for researchis unwavering, which rules out commercial pack-ages. Perhaps the most well known open-sourcefluid-flow software is OpenFOAM, so we set outto replicate our published results with this code.A more specialist open-source code is IBAMR, aproject born at New York University that has con-tinued development for a decade. And finally,our own group developed a new code, imple-menting the same solution method we had be-fore, but providing parallel computing via therenowned PETSc library. We embarked on a fullreplication study of our previous work, usingthree new fluid-flow codes.
This is the story of what happened next: threeyears of dedicated work that encountered adozen ways that things can go wrong, conqueredone after another, to arrive finally at (approxi-mately) the same findings and a whole new un-derstanding of what it means to do reproducibleresearch in computational fluid dynamics.
Fluid-flow solvers we used:
cuIBM Used for our original study,2 this code is written in C CUDA to exploit GPU hardware, but is serial on CPU. Ituses the NVIDIA Cusp library for solving sparse linear systems on GPU. https://github.com/barbagroup/cuIBMOpenFOAM A free and open-source CFD package that includes a suite of numerical solvers. The core discretizationscheme is a finite-volume method applied on mesh cells of arbitrary shape. http://www.openfoam.orgIBAMR A parallel code using the immersed boundary method on Cartesian meshes, with adaptive mesh refinement.https://github.com/ibamr/ibamrPetIBM Our own re-implementation of cuIBM, but for distributed-memory parallel systems. It uses the PETSc library forsolving sparse linear systems in parallel. https://github.com/barbagroup/PetIBM
Story 1: Meshing and boundary con-ditions can ruin everything
Generating good discretization meshes is proba-bly the most vexing chore of computational fluiddynamics. And stipulating boundary conditionson the edge of a discretization mesh takes somenerve, too. Our first attempts at a full replica-tion study of the 2D snake aerodynamics withIcoFOAM, the incompressible laminar Navier-Stokes solver of OpenFOAM, showed us just howvexing and unnerving this can be.
OpenFOAM can take various types of dis-cretization mesh as input. One popular meshgenerator is called GMSH: it produces trianglesthat are as fine as you want them near the body,while getting coarser as the mesh points are far-ther away. Already, we encounter a problem:how to create a mesh of triangles that gives acomparable resolution to that obtained with ouroriginal structured Cartesian mesh? After ded-icated effort (see supplementary material), weproduced the best mesh we could that matchesour previous study in the finest cell width nearthe body. But when using this mesh to solvethe fluid flow around the snake geometry, we gotspurious specks of high vorticity in places wherethere shouldnt be any (Figure 1). Despite the factthat the mesh passed the quality checks of Open-FOAM, these unphysical vortices appeared forany flow Reynolds number or body angle of at-tack we tried, although they were not responsiblefor the simulations to blow up. Finally, we gaveup with the (popular) GMSH and tried anothermesh generator: SnappyHexMesh. Success! Nounphysical patches in the vorticity field this time.But another problem persisted: after the wakevortices hit the edge of the computational domainin the downstream side, a nasty back pressure ap-peared there and started propagating to the in-side of the domain (Figure 2). This situation is
also unphysical, and we were certain there was aproblem with the chosen outflow boundary con-dition in OpenFOAM, but did not find any wayto stipulate another, more appropriate boundarycondition. We used a zero-gradient condition forthe pressure at the outlet (and tried several otherpossibilities), which we found was a widespreadchoice in the examples and documentation ofOpenFOAM. After months, one typing mistakewhen launching a run from the command linemade OpenFOAM print out the set of availableboundary conditions, and we found that an advec-tive condition was available that could solve ourproblem (all this time, we were looking for a con-vective condition, which is just another name forthe same thing). Finally, simulations with Open-FOAM were looking correctand happily, themain feature of the aerodynamics was replicated:an enhanced lift coefficient at 35 degrees angle-of-attack (Figure 3). But not all is perfect. The timesignatures of lift and drag coefficient do show dif-ferences between our IcoFOAM calculation andthe original published ones (Figure 4). The keyfinding uses an average lift coefficient, calculatedwith data in a time range that is reasonable butarbitrary. Although the average force coefficientsmatch (within
Figure 1: Vorticity field after 52time units of flow simulation withIcoFOAM for a snakes section withangle-of-attack 35 degrees andReynolds number 2000. We cre-ated a triangular mesh (about 700ktriangles) with the free softwareGMSH. Notice the patch of spuri-ous vorticity at the center-bottom ofthe domain.
Figure 2: Pressure field after 52(top) and 53 (bottom) time units offlow simulation with IcoFOAM forsnake section with angle-of-attack35 degrees and Reynolds number2000. The simulation crashed afterabout 62 time-units because of theback pressure at the outlet bound-ary.
25o 30o 35o 40o
angle of attack
Re = 1000
Krishnan et al. (2014)
Re = 2000
Krishnan et al. (2014)
Figure 3: Time-averaged drag (top) and lift (bottom) coeffi-cients as function of the snakes angle-of-attack for Reynoldsnumbers 1000 and 2000. We averaged all IcoFOAM force co-efficients between 32 and 64 time-units of flow-simulation aswe have done in our previous study.
grids are complex geometrical objects. Two un-structured meshes built with the same parame-ters will not be exactly the same, even. The mesh-generation procedures are not necessarily deter-ministic, and regularly produce bad triangles thatneed to be repaired. The complications of build-ing a good quality mesh is one of the reasons someprefer immersed boundary methods!
Story 2: Other researchers open-source codes come with traps
Open-source research software can often bepoorly documented and unsupported, and on oc-casion it can even be an unreadable mess. But inthis case, we are in luck. IBAMR is a solid pieceof software, well documented, and you can evenget swift response from the authors via the topi-cal online forum. Still, we ran against an obscuretrick of the trade that changed our results