The Adaptive TreePM: an adaptive resolution code for cosmological N-body simulations

Mon. Not. R. Astron. Soc. 396, 2211–2227 (2009) doi:10.1111/j.1365-2966.2009.14880.x

The Adaptive TreePM: an adaptive resolution code for cosmologicalN-body simulations

J. S. Bagla� and Nishikanta Khandai�Harish-Chandra Research Institute, Chhatnag Road, Jhunsi, Allahabad 211019, India

Accepted 2009 April 5. Received 2009 March 9; in original form 2009 January 20

ABSTRACTCosmological N-body simulations are used for a variety of applications. Indeed progress inthe study of large-scale structures and galaxy formation would have been very limited withoutthis tool. For nearly 20 yr the limitations imposed by computing power forced simulatorsto ignore some of the basic requirements for modelling gravitational instability. One of thelimitations of most cosmological codes has been the use of a force softening length that is muchsmaller than the typical interparticle separation. This leads to departures from collisionlessevolution that is desired in these simulations. We propose a particle-based method with anadaptive resolution where the force softening length is reduced in high-density regions whileensuring that it remains well above the local interparticle separation. The method, called theAdaptive TreePM (ATreePM), is based on the TreePM code. We present the mathematicalmodel and an implementation of this code, and demonstrate that the results converge over arange of options for parameters introduced in generalizing the code from the TreePM code.We explicitly demonstrate collisionless evolution in collapse of an oblique plane wave. Wecompare the code with the fixed resolution TreePM code and also an implementation thatmimics adaptive mesh refinement methods and comment on the agreement and disagreementsin the results. We find that in most respects the ATreePM code performs at least as well as thefixed resolution TreePM in highly overdense regions, from clustering and number density ofhaloes to internal dynamics of haloes. We also show that the adaptive code is faster than thecorresponding high-resolution TreePM code.

Key words: gravitation – methods: N-body simulations – large-scale structure of Universe.

1 IN T RO D U C T I O N

Large-scale structures traced by galaxies are believed to haveformed by amplification of small perturbations (Peebles 1980;Peacock 1999; Bernardeau et al. 2002; Padmanabhan 2002). Galax-ies are highly overdense systems, matter density ρ in galaxies isthousands of times larger than the average density ρ in the universe.Typical density contrast (δ ≡ ρ/ρ − 1) in matter at these scalesin the early universe was much smaller than unity. Thus the prob-lem of galaxy formation and the large-scale distribution of galaxiesrequire an understanding of the evolution of density perturbationsfrom small initial values to the large values we encounter today.

Initial density perturbations were present at all scales that havebeen observed (Percival et al. 2007; Spergel et al. 2007). The equa-tions that describe the evolution of density perturbations in an ex-panding universe have been known for several decades (Peebles1974) and these are easy to solve when the amplitude of pertur-

�E-mail: [email protected] (JSB); [email protected] (NK)

bations is small. Once density contrast at relevant scales becomescomparable to unity, perturbations becomes non-linear and cou-pling with perturbations at other scales cannot be ignored. Theequation for evolution of density perturbations cannot be solvedfor generic initial conditions in this regime. N-body simulations(e.g. see Efstathiou et al. 1985; Bagla & Padmanabhan 1997a;Bertschinger 1998; Bagla 2005; Dolag et al. 2008) are often usedto study the evolution in this regime. Alternative approaches can beused if one requires only a limited amount of information and insuch a case either quasi-linear approximation schemes (Zel’Dovich1970; Gurbatov, Saichev & Shandarin 1989; Matarrese et al. 1992;Brainerd, Scherrer & Villumsen 1993; Bagla & Padmanabhan 1994;Sahni & Coles 1995; Hui & Bertschinger 1996; Bernardeau et al.2002) or scaling relations (Davis & Peebles 1977; Hamilton et al.1991; Nityananda & Padmanabhan 1994; Peacock & Dodds 1994;Peacock & Dodds 1996; Jain, Mo & White 1995; Padmanabhan1996; Padmanabhan et al. 1996; Ma 1998; Kanekar 2000; Smithet al. 2003) suffice. However, even the approximation schemes andscaling relations must be compared and calibrated with simulationsbefore these can be used with confidence.

C© 2009 The Authors. Journal compilation C© 2009 RAS

2212 J. S. Bagla and N. Khandai

Last three decades have seen a rapid development of techniquesand computing power for cosmological simulations and the re-sults of these simulations have provided valuable insight into thestudy of structure formation. The state of the art simulations usedless than 105 particles two decades ago (Efstathiou et al. 1988)and if the improvement had been due only to computing powerthen the largest simulation possible today should have been around109 particles, whereas the largest simulations done till date usedmore than 1010 particles (Springel et al. 2005). Evidently, develop-ment of new methods and optimizations has also played a signif-icant role in the evolution of simulation studies (Efstathiou et al.1985; Barnes & Hut 1986; Greengard & Rokhlin 1987; Bouchet &Hernquist 1988; Jernigan & Porter 1989; Hernquist 1990; Makino1990, 1991; Couchman 1991; Hernquist, Bouchet & Suto 1991;Ebisuzaki et al. 1993; Theuns 1994; Brieu, Summers & Ostriker1995; Suisalu & Saar 1995; Xu 1995; Dubinski 1996; Kravtsov,Klypin & Khokhlov 1997; Macfarland et al. 1998; Bode, Ostriker& Xu 2000; Brieu & Evrard 2000; Dehnen 2000; Kawai & Makino2001; Knebe, Green & Binney 2001; Springel, Yoshida & White2001; Bagla 2002; Dehnen 2002; Makino 2002; Bagla & Ray 2003;Bode & Ostriker 2003; Makino et al. 2003; Dubinski et al. 2004;Makino 2004; Ray & Bagla 2004; Wadsley, Stadel & Quinn 2004;Merz, Pen & Trac 2005; Springel 2005; Yoshikawa & Fukushige2005; Thacker & Couchman 2006; Khandai & Bagla 2009). Alongthe way, code developers have also successfully met the challengeposed by the emergence of distributed parallel programming.

In modelling gravitational clustering, cosmological N-bodycodes should ensure the following.

(i) The universe does not have a boundary. Therefore, cosmolog-ical simulations need to be run with periodic boundary conditions.1

The simulation volume should be large enough for the effects ofmissing modes to be small (Bagla & Ray 2005; Bagla & Prasad2006; Power & Knebe 2006; Prasad 2007; Bagla, Prasad & Khandai2009).

(ii) The mass of each particle in simulations should be muchsmaller than mass scales of interest in the simulation output. Thisis to ensure adequate mass resolution.

(iii) Each particle in an N-body simulation represents a very largenumber of particles/objects in the universe. Thus we must ensurethat pairwise interaction of N-body particles is softened at scalescomparable with the local interparticle separation. If this is notensured then the resulting two-body collisions introduce errors inthe resulting distribution of particles (Splinter et al. 1998; Binney& Knebe 2002).

In spite of the vast improvement in computing power, simulatorshave often had to compromise on one or more of these points. Of-ten, errors also creep in due to the approximate methods used forsolving for force in simulations. A large fraction of current cosmo-logical N-body codes suffers from collisionality or force biasing.Force is biased when softening lengths ε are much larger than thelocal mean interparticle separation, rij . Whereas a complementaryeffect, collisions, occur whenever ε � rij . The reader is referredto Dehnen (2001) for a detailed discussion on these two effects.Codes that adapt their softening lengths to local densities generallyare mostly of the adaptive mesh refinement (AMR) type. These usea grid for solving the Poisson equation and often have anisotropiesin force at the scales comparable to the size of a grid cell. Codes

1 An exception is simulations of large spherical volumes that do not suffersignificant deformation during the course of evolution.

which use fixed softening lengths are not entirely collisionless, andin highly overdense regions biasing of force also exists. The TPM (Xu1995; Bode et al. 2000; Bode & Ostriker 2003) is a particle-basedcode with a one step adaptive resolution. However, in this case theuse of the unmodified Particle-Mesh (PM) approach for computingthe long-range force introduces significant errors at scales compara-ble to the grid. In this work we describe a code which addresses allthree issues of force anisotropy, collisionality and force biasing byemploying an adaptive softening formalism with the hybrid TreePMcode.

The choice of the optimal softening length has been discussedat length (Merritt 1996; Athanassoula et al. 2000; Dehnen 2001;Price & Monaghan 2007). These studies were carried out in thecontext of isolated haloes in dynamical equilibrium, e.g. Plummerand Hernquist profiles, therefore, errors could be defined clearly. Itis also possible to compute and compare various physical quantitieswith analytical expressions derived from the distribution function(DF). Dehnen (2001) derived analytical expressions for errors inthe context of these profiles. This work suggested that the optimalsoftening length must adapt to the local interparticle separation asa function of space and time. Price & Monaghan (2007) have de-veloped an energy–momentum conserving formalism with adaptivesoftening and demonstrated that it was superior to fixed softening.

The evolution of perturbations at small scales depends stronglyon the mass and force resolution. High force resolutions can leadto better modelling of dense haloes, but gives rise to two bodycollisions in regions where the softening length is smaller than thelocal interparticle separation (Splinter et al. 1998; Binney & Knebe2002; Binney 2004; Diemand et al. 2004; El-Zant 2006; Romeoet al. 2008). As all particles in very high density regions go throughsuch a phase during evolution, any errors arising due to two-bodycollisions can potentially affect the structure of high-density haloesthat form. A high force resolution without a corresponding massresolution can also give misleading results as we cannot probeshapes of collapsed objects (Kuhlman, Melott & Shandarin 1996).In addition, discreteness and stochasticity also limit our ability tomeasure physical quantities in simulations, and these too need tobe understood properly (Romeo et al. 2008; Thiebaut et al. 2008).In all such cases, the errors in modelling are large at small scales.It is important to understand how such errors may spread to largerscales and affect physical quantities.

This work is organized as follows. In Section 2 we describe theformalism for adaptive softening in a cosmological N-body code.In Section 3 we review the TreePM method, which forms the back-bone of our gravity solver and describe how adaptive softening isimplemented with it. We briefly discuss the performance charac-teristics of Adaptive TreePM (ATreePM) in Section 4. We discussvalidation of the ATreePM code in Section 5, and we conclude inSection 6.

2 A DA PTI VE FORCE SOFTENI NG

2.1 Formalism

The aim of any collisionless N-body code is to self-consistentlyevolve the phase-space DF f (r , v, t) under its own gravitationalforce field F(r , t):

∂t f + (∂rf ) · v + (∂vf ) · F = 0, (1)

F(r, t) = −G

∫ ∫d3r ′d3v

r − r ′

|r − r ′|3 f (r ′, v, t). (2)

C© 2009 The Authors. Journal compilation C© 2009 RAS, MNRAS 396, 2211–2227

Adaptive TreePM 2213

The approach that one takes is to sample the DF, by N phase-spacepoints, {r i , vi}t=ti

, at initial time t = t i. Liouville’s theorem thenstates that evolving the trajectories of these points to any time t > t i

will be a representation of the DF at that time. Since the system is acollisionless one, one has to suppress artificial two-body collisionsarising out of interactions between particles which were generatedfor sampling the density field. One therefore assigns a finite size toN-body particles which ensures softening of force at small scales,instead of assuming these to be point particles. The density fieldwhen sampled by point particles,

ρ(r) =∫

d3vf (r, v, t) ≡N∑

j=1

mjδ3D

(r − r j

)(3)

is now smoothed at small scales if we assign a finite size ε to everyparticle:

ρ(r) =∫

d3vf (r, v, t) ≡N∑

j=1

mjW(|r − r j |, ε

), (4)

where W (u, ε) is known as the smoothing kernel and we have as-sumed that particles are spherical in shape. Here mj is the mass ofthe jth particle. We can now integrate the Poisson equation to ob-tain the expression for the kernel for computing force and potential.Both the quantities are softened at scales below the softening lengthε. We choose to work with the cubic spline kernel (Monaghan &Lattanzio 1985) whose expression is given below. Complete ex-pressions for the potential and force are given in Appendix A (seeequations A1 and A2):

W (u, ε) = 8

πε3

⎧⎪⎨⎪⎩1 − 6 (u/ε)

2 + 6 (u/ε)3, 0 ≤ u/ε < 0.5,

2 [1 − (u/ε)]3, 0.5 ≤ u/ε < 1.0,

0, 1.0 ≤ u/ε.

(5)

In the context of individual softening lengths the density at thelocation of the ith particle is given by

ρ(ri) =N∑

j=1

mjW (rij , εi) =nn∑

j=1

mjW (rij , εi), (6)

where i, j indicate the particle indices, rij = |r i − r j| and εi ≡ε(r i). The summation in principle can be extended up to infinityif the kernel is infinite in extent (e.g. Plummer, Gaussian kernels).However, since such kernels tend to bias the force (Dehnen 2001;Price & Monaghan 2007) we will work only with those kernelswhich have compact support, in particular the cubic spline kernel.Such kernels ensure that the force is Newtonian beyond the soft-ening scale. nn is the number of the nearest neighbours within εi

for the particle i. In the discussion that follows we assume that thisnumber is fixed for every particle and sets the value of the softeninglength εi. We implicitly assume it in our summation. Integrating thePoisson equation we obtain the Green’s function for the potentialφij ≡ φ(rij, εi), where the functional form for φ is given in AppendixA (see equations A1 and A2).

With the introduction of individual softening lengths for particles,the symmetry of the potential is lost and momentum conservationis violated. Since ε(r) is now a local quantity, the equation of mo-tion (EOM) is incomplete if one takes the expression of force withthe fixed softening length ε replacing it by a local softening lengthε(r). Hence energy conservation also gets violated. This is becausethe force is derived from the potential and with the introductionof a local softening length ε(r) in the potential, the gradient mustalso act on ε(r) giving us an extra term. Traditionally this grad-ε

(∇ε) term has been ignored since in typical applications these werefound to be subdominant when compared to the usual force (Gin-gold & Monaghan 1982; Evrard 1988; Hernquist & Barnes 1990;Monaghan 1992). It has been shown recently (Price & Monaghan2007) that this term plays an important role in N-body simulations.We study the impact of ignoring this term in cosmological simula-tions.

A remedy for momentum non-conservation is to use a sym-metrized softening length

εij = 1

2(εi + εj ), (7)

and plug it into the expression for density to rederive a symmetrizedexpression for the potential as φ(rij, εij). This prescription changesthe softening length and hence the neighbour list, which one hasto recompute. Another disadvantage is that with this prescriptionfor symmetrization, the number of neighbours is not fixed for everyparticle and hence errors in all smoothed estimates are not the samefor every particle. An alternate method (Hernquist & Katz 1989) isto symmetrize the kernel itself:

Wij = Wji = 1

2

[W

(rij , εj

) + W(rij , εi

)],

φij = 1

2

[φ

(rij , εj

) + φ(rij , εi

)]. (8)

The total potential is thus

�tot = 1

2

N∑i,j

φij . (9)

We can use this to write a Lagrangian which is manifestly symmetricand this ensures momentum conservation. Energy is conserved onlyif the ∇ε term is retained in the EOM:

L =N∑i

1

2miv

2i − G

2

N∑i,j

mimj φij . (10)

The EOM can be derived with this Lagrangian (the reader isreferred to Price & Monaghan 2007 for details):

dvi

dt= − G

∑j

mj φ′ij

r i − rj

|r i − rj |

− G

2

∑j

mj

[ζi

i

∂Wij (εi)

∂r i

+ ζj

j

∂Wij

(εj

)∂r i

], (11)

where the first term is the standard Newtonian force term (we referto it as the ∇φ term). The second is the energy conserving ∇ε termwhich would be zero for fixed softening. Notice that all terms areantisymmetric in i, j and hence the total momentum is conserved.Here ζ and are

ζi ≡ ∂εi

∂ρi

∑j

mj

∂φ(rij , εi

)∂εi

, (12)

i =[

1 − ∂εi

∂ρi

∑j

mj

∂Wij (εi)

∂εi

]. (13)

Expressions for ∂W/∂r , ∂W/∂ε and ∂φ/∂ε are given in AppendixA (see A3, A4 and A5). As εi ∝ ρ−(1/3)

i we have ∂εi/∂ρ i =−(εi/3ρ i). The term term ensures that the EOM is accurate to allorders in ε (Springel & Hernquist 2002).



3 TH E A DA P T I V E TreePM ME T H O D

3.1 The TreePM method

The TreePM algorithm (Bagla 2002; Bagla & Ray 2003) is a hybridN-body method which improves the accuracy and performance ofthe Barnes–Hut (BH) tree method (Barnes & Hut 1986) by com-bining it with the PM method (Klypin & Shandarin 1983; Miller1983; Bouchet, Adam & Pellat 1985; Bouchet & Kandrup 1985;Hockney & Eastwood 1988; Bagla & Padmanabhan 1997a; Merz,Pen & Trac 2005). The TreePM method explicitly breaks the po-tential into a short-range and a long-range component at a scale rs.The PM method is used to calculate long-range force and the short-range force is computed using the BH tree method. Use of the BHtree for short-range force calculation enhances the force resolutionas compared to the PM method.

The gravitational force is divided into a long-range and a short-range part using partitioning of unity in the Poisson equation:

ϕk = −4πGρk

k2,

ϕk = ϕlrk + ϕsr

k ,

ϕlrk = −4πGρk

k2exp

(−k2r2s

),

(14)

ϕsrk = −4πGρk

k2

[1 − exp

(−k2r2s

)]. (15)

Here ϕsrk and ϕlr

k are the short-range and long-range potentials inFourier space. ρ is the density, G is the gravitational coupling con-stant and rs is the scale at which the splitting of the potential isdone. It has been shown that this particular split between the long-range and the short-range force is optimal amongst a large classof suitable functional forms (Bagla & Ray 2003). The long-rangeforce is computed in Fourier space with the PM method and theshort-range force is computed in real space with the Tree method.The short-range potential and force in real space are

ϕsr(r, ε) = Gmφ (r, ε) erfc

(r

2rs

), (16)

Fsr (r, ε) = Gm f (r, ε) C

(r

rs

), (17)

C

(r

rs

)=

[erfc

(r

2rs

)+ r

rs√

πexp

(− r2

4r2s

)]. (18)

Here erfc is the complementary error function. φ(r , ε) and f (r , ε)are the usual potential and force kernels, respectively. C(r/r s) mod-ifies the softened Newtonian force kernel f (r , ε) to the short-rangeforce Fsr(r , ε). The expression for φ(r , ε) and f (r , ε) dependson the kernel W (r , ε) used for smoothing and are given for cubicspline in Appendix A. We find that tabulating erfc (u) and C(u)and using interpolation to compute these functions is much moreeffective than calculating them every time. The short range force isbelow 1 per cent of the total force at r ≥ 5r s. The short range forceis therefore computed within a sphere of radius rcut � 5r s using theBH tree method.

The long-range force falls below 1 per cent of the total forcefor r ≤ 0.5r s. Thus the choice of softening length does not affectthe long-range force in a significant manner as long as the forcesoftening is done with a kernel that has a compact support and thesoftening length is below 0.5r s.

The BH tree structure is built out of cells and particles. Cellsmay contain smaller cells (subcells) within them. Subcells can haveeven smaller cells within them, or they can contain a particle. Inthree dimensions, each cubic cell is divided into eight cubic subcells.Cells, as structures, have attributes like total mass, location of centreof mass and pointers to subcells. Particles, on the other hand, havethe usual attributes: position, velocity and mass.

Force on a particle is computed by adding contribution of otherparticles or of cells. A cell that is sufficiently far away can beconsidered as a single entity and we can add the force due to the totalmass contained in the cell from its centre of mass. If the cell is notsufficiently far away then we must consider its constituents, subcellsand particles. Whether a cell can be accepted as a single entity forforce calculation is decided by the cell acceptance criterion (CAC).We compute the ratio of the size of the cell Lcell and the distance rfrom the particle in question to its centre of mass and compare itwith a threshold value

θ = Lcell

r≤ θc. (19)

The error in force increases with θ c. Poor choice of θ c can leadto significant errors (Salmon & Warren 1994). Many different ap-proaches have been tried for the CAC in order to minimize error aswell as CPU time usage (Salmon & Warren 1994; Springel et al.2001). The tree code gains over direct summation as the numberof contributions to the force is much smaller than the number ofparticles.

The TreePM method is characterized therefore by three parame-ters, r s, rcut and θ c. For a discussion on the optimum choice of theseparameters the reader is referred to Bagla & Ray (2003). For allour tests we choose conservative values r s = 1.0, rcut = 5.2r s andθ c = 0.3 which give errors below 1 per cent in force. All lengthsare specified in units of the PM grid.

3.2 The Adaptive TreePM method

We choose to implement adaptive softening with a modified TreePMcode (Khandai & Bagla 2009), which incorporates Barnes opti-mization using groups (Barnes 1990; Makino 1991; Yoshikawa& Fukushige 2005) into the TreePM code. In principle one canalso incorporate a similar formalism for tree codes (Springel et al.2001), P3M codes (Couchman 1991; Couchman, Thomas & Pearce1995) and other variants like TPM (Xu 1995), TreePM (Bagla 2002;Springel 2005) and GOTPM (Dubinski et al. 2004). We shall discussone advantage of using the TreePM code with the group optimiza-tions below.

3.2.1 Estimating the softening length

Our first task is to get an estimate of the softening length for eachparticle. A natural way to extract a local length scale uses thenumerical value of local density. The local number density is relatedto the softening length as

ε(r) ∼[

nn

n (r)

]1/3

, (20)

n(r) is the number density at the location of the particle and weassume all particles have the same mass. Here, nn is a referencenumber and we take it to be the number of neighbours used forestimation of the number density. The above equation is implicitand can be solved iteratively, see e.g. Springel & Hernquist (2002);Springel (2005); Price & Monaghan (2007) for details. Price &



Monaghan (2007) have shown that errors are not very sensitive tothe exact value of nn. We choose nn = 32 in our simulations, andalso comment on variation in results with this choice.

We are using the formalism developed by Price & Monaghan(2007) for achieving an adaptive resolution in gravitational interac-tions of particles. As the formalism was developed in the contextof smoothed particle hydrodynamics (SPH) codes, and some ofthe quantities required can be computed naturally using the SPHmethod, the same was used even though the gravitational interac-tion is completely collisionless. We follow a similar implementationand use methods commonly used in SPH simulations, even thoughthere are no hydrodynamical effects present in the gravitationalinteraction being studied here. For an overview of SPH methods,please see Gingold & Monaghan (1977) and Lucy (1977). The SPHmethods assign values for functions like the density to particles byaveraging over nearest neighbours. The list of nearest neighbours isan essential requirement for computing anything using these meth-ods. All quantities (ρ, ∇ε, , ζ etc.) can be computed at runtimewith this neighbour list once we have converged to a value for ε bysolving equation (20). We compute the neighbour list using linkedlists (Hernquist & Katz 1989).

We put bounds on the maximum and minimum softening lengths,εmax and εmin. A maximum bound is required so that force softeningis restricted to the short-range force only, we choose εmax = r s/2 inorder to ensure that the long-range force is of order 1 per cent of thetotal force (or smaller) at scales where force softening is important.This ensures that any errors arising from non-modification of thelong-range force are smaller than 1 per cent, if one puts a lowerthreshold on the maximum allowed error then the scale εmax hasto be lowered correspondingly. Alternatively, one can work witha larger rs and then it becomes possible to allow a larger εmax. Alower bound is also required for the reason that we do not want afew isolated highly overdense regions to dominate the CPU timerequirements. The value of the lower bound must correspond todensities that are much higher than highest overdensities of interestin the simulation. We set this lower bound εmin = r s/1000, whichcorresponds to ρ ∼ 1010ρ. The lower bound however is not criticalto the structure of the code and may be omitted.

In our implementation a neighbour search is carried out onlyup to εmax. Particles which do not have nn neighbours within εmax

are assigned εi = εmax and the spherical top hat (STH) densityis assigned with the number of neighbours within εmax. For theseparticles we assign = 1 and ζ = 0, which makes their ∇εi =0. A similar assignment is carried out for particles which haveεi ≤ εmin. The ∇ε term is calculated before the short-range force sothat the individual softening lengths needed for short-range forcecalculation are also assigned in the process.

3.2.2 Memory requirements

Even though we use two separate data structures, namely linked listsfor ∇ε and tree for ∇φ in order to compute the total short-rangeforce, additional memory requirements compared to TreePM areminimal: we require one additional array for storing the softeninglengths. The ∇ε term does not require an additional array since it isa component of the short-range force and it can be computed at runtime. This is because the two data structures are never required at thesame time. We require specification of the largest force softeninglength in a given cell (see the subsection on the CAC below). Thisamounts to a single precision array of the same size as the numberof cells in the tree.

An advantage of using an analytical splitting of force, in themanner TreePM does, is that computation of short-range force doesnot need global data structures. For example one can geometricallydivide regions into smaller regions and construct local trees andlinked lists to them (just like one would go about doing it in adistributed code) and iterate through these regions for computingshort-range force instead of constructing one global data structurefor the entire simulation volume for computing the short-range force(Dubinski et al. 2004). This reduces memory usage significantly andthe dominant part is taken up by the arrays required for computingthe long-range PM force.

3.2.3 Time-stepping criterion and integration

We have implemented a hierarchical time integrator similar to thatused in GADGET-2 (Springel 2005), in which particle trajectoriesare integrated with individual time-steps and synchronized with thelargest time-step. As we allow the block time-step2 to vary withtime, we work with the so-called KDK (kick-drift-kick) approachin which velocities are updated in two half steps whereas position isupdated in a full step. It can be shown that with a variable time-step,KDK performs better than DKD (drift-kick-drift; see Springel 2005for details). We give separate PM (long range, global) kicks andtree (short range, individual) kicks.3 The block time-step �t

pmis

determined by the particle which has the maximum PM accelerationa

pm

max:

�tpm = δt

(εmax

apmmax

)1/2

. (21)

Here δt is the dimensionless accuracy parameter. In our implemen-tation of the hierarchy of time-steps, the smaller time-steps differ byan integer power (n) of 2 from the largest, block time-step. An arrayis then used to store the value n which determines the time-step ofthe particle. Individual time-steps �ti are first calculated,

�t sri = δt

(εi

asri

)1/2

, (22)

and then the appropriate hierarchy n is chosen depending on thisvalue. Here asr

i and εi are the modulus of the individual short-range acceleration (sum of the ∇φ and ∇ε terms) and the softeninglengths, respectively. TreePM has a similar time-stepping criterionwith εi replace by ε.

The code drifts all the particles with the smallest time-step tothe next time, where a force computation is done for particles thatrequire an update of velocity (kick). However, the neighbour listand individual softening lengths εi are computed for all particles atevery small time-step. This is because, even though some particlesdo not require a velocity update, their neighbours might require onefor which they would contribute through their updated softeninglengths. The ∇ε term, however, can be computed only for thoseparticles requiring a velocity update.

Within a given block time-step, the smaller time-steps are con-stant for a given particle. The time-step changes across block time-step and this brings in inaccuracies in evolution of trajectories. Itis possible, in principle, to ensure that the second-order accuracy ismaintained here. However, we find that the time-steps for particleschange very slowly and this change does not affect trajectories in asignificant manner.

2 Same as the largest time-step.3 Tree kick includes the contribution of the ∇ε term.



Figure 1. Wall clock timing for TreePM (left) as a function of ε and ATreePM (right) with ∇ε (solid) and without ∇ε (dashed) as a function of nn.

The Courant condition is satisfied for the choice of δt we use.Indeed, we chose δt by requiring that two particles in a highlyeccentric orbit around each other maintain the trajectory correctlyfor tens of orbits.

3.2.4 Cell acceptance criterion for ATreePM

The adaptive force resolution formalism requires us to symmetrizeforce between particles that are separated by a distance smaller thanthe larger of the two softening lengths. Without this, the momentumconservation cannot be ensured. For pairs of particles separated bylarger distances, there is no need to explicitly symmetrize forceas there is no dependence on the softening length at these scales.4

Thus the CAC needs to be changed within the tree part of the codeto ensure that for pairs of particles separated by the critical dis-tance, forces are computed through pairwise PP interaction. In ourimplementation, along with particles, cells (and hence groups) areassigned softening lengths corresponding to the particle containedwithin the cell that has the largest softening length. The CAC thatis modified when we go from the TreePM to the modified TreePM(Khandai & Bagla 2009) has to be changed again to take into ac-count the largest softening lengths for particles within cells andgroups whose interaction is to be computed:(

Lcell + εmax

cell

)(rgcm − Lgroup − ε

max

group

) ≤ θc, (23)

εmax

cell, ε

max

groupare the softening lengths of the cell and group (in the sense

explained above), respectively. rgcm is the distance separating the

4 As an aside we would like to note that the Tree method does not conservemomentum explicitly. This is because the tree traversal approximates theforce due to pairwise interactions, and in the process the pairwise sym-metry is lost. In the modified Tree method (Barnes 1990; Makino 1991;Yoshikawa & Fukushige 2005; Khandai & Bagla 2009) explicit pairwiseforce is computed for particles within each group. Particles within a grouphave a common interaction list for force due to particles outside the group.Exact pairwise particle–particle (PP) force is computed explicitly for intra-group particles and we have a pairwise symmetry for this component, butfor interaction with particles outside the group there is no explicit pairwisesymmetry and hence no explicit momentum conservation.

centres of mass of the group and the cell. Lcell is the size of the cell,and Lgroup is the distance to the furthest particle from the centre ofmass of the group. This CAC ensures that the interaction of particlesseparated by less than the softening length is computed in a directpairwise manner and hence can be explicitly symmetrized. Theseare also the pairs for which the ∇ε term needs to be computed.

4 PE R F O R M A N C E C H A R AC T E R I S T I C S

4.1 Timing

We now look at the wall clock time as a measure of performancebetween different codes. We studied evolution of a power-law modelwith n = −1 using 1283 particles up to the stage where the scale ofnon-linearity is 6 grid lengths. More details of the run are given inthe section on validation of the ATreePM code. Fig. 1 shows wallclock time as a function of softening length ε for TreePM (left-handpanel) and the wall clock time as a function of nn for the ATreePMcodes.

We can qualitatively understand the slope for the TreePM curveby looking at the equivalent time-stepping criterion as equation (22)for TreePM:

�t sri = δt

(ε

asri

)1/2

. (24)

From here the naive expectation is that the time taken should scaleas ε−0.5. The slope of the curve is in the range −0.55 to −0.65. Thereason for this small deviation lies in our use of a hierarchy of time-steps, where trajectories of all the particles are not updated at everytime-step. However, positions of particles are updated at every time-step and this operation as well as those related to creating the treestructure at every step adds an overhead. This overhead becomesmore and more important at small ε where we have many morelevels of hierarchy realized in a simulation. This leads to steepeningof the curve from the simple expectation given above.

For the ATreePM, we expect softening lengths to be larger forlarger nn. On the other hand, a larger nn implies a larger neighbourlist and the time taken for setting up the neighbour list increases. Thesecond effect is the dominant one and we see that the time taken forATreePM increases with nn. We see that time taken by both variantsof ATreePM is similar for nn = 32 and 48. This indicates that the



Figure 2. Left: cumulative distribution of softening lengths at the final epoch of an N-body simulation (see Fig. 8, bottom left). Right: ASE for ATreePM with∇ε (solid line) and ATreePM without ∇ε (dashed line) as a function of nn.

time taken for calculation of the ∇ε term is negligible. There is adifference between the timing for nn = 16, as the code with the∇ε does not evolve the system correctly: this can be seen in allthe indicators like the amplitude of clustering, mass function etc.,presented in the next section. We see that TreePM with ε = 0.025takes 50 per cent more time than ATreePM with nn = 32, whereasthe time taken are comparable with ATreePM with nn = 48.

Since ε is assigned by hand in TreePM, the use of a small soften-ing length means a considerable number of particles obtain a smalltime-step even though the acceleration for these particles is smalland does not vary rapidly with time. In a hierarchical integrator thatwe use, this means that force computation is done more often withina block time-step. In ATreePM on the other hand a small softeningis only assigned to particles in overdense regions. This saves timein the underdense and not so overdense regions and the ATreePMcode devotes more time to evolving trajectories in highly overdenseregions. This is illustrated in Fig. 2 where we have plotted the cu-mulative distribution of softening lengths at the final epoch in oneof the simulations used here (see lower left-hand panel of Fig. 8).The softening length was computed here for nn = 32. We find thatonly 5 per cent of the particles have a softening length smaller thanthe smallest softening length ε = 0.025 used for the fixed resolutionsimulations. Even at this epoch where highly non-linear clustering isseen, nearly half the particles have a softening length correspondingto the maximum value of 0.5. This shows how we are able to evolvethe system in an ATreePM simulation with a lower computationalcost while resolving highly overdense regions.

4.2 Errors

In this section we discuss the dependence of errors on nn. In cos-mological simulations it is difficult to define errors when softeninglengths are varied due to the lack of a reference set-up. Nevertheless,we can choose the optimal softening length such that one minimizesthe globally averaged fluctuation in force as the softening lengthsare varied. Following Price & Monaghan (2007), we define averagesquare error (ASE):

ASE(nn) = C

N

N∑i

∣∣ f i(nn) − f i(nn + �nn)∣∣2

, (25)

N is the total number of particles. C is a normalization constantwhich is taken as 1/f 2

max, with f max being the largest value of forcein either runs. �nn is the change in nn and we choose this to be8. With other values of �nn the overall behaviour of the error plotremains qualitatively the same. Fig. 2 shows ASE as function of nn

for ATreePM with ∇ε (solid line) and ATreePM without ∇ε (dashedline). These were computed for the same clustered distribution ofparticles. The qualitative behaviour of ASE here is same as seen inPrice & Monaghan (2007). At small nn ATreePM with ∇ε has largererrors than ATreePM without ∇ε. This is also reflected in the poorevolution of density fluctuations with nn = 16 for the ATreePM with∇ε, discussed in the next section. Both variants have a minima, thevalue of error at the minima being lower for ATreePM with the ∇ε

term. Another interesting feature is that the region where errors aresmall is fairly broad for the ATreePM with the ∇ε term. For largernn the error increases sharply for ATreePM without the ∇ε term.Thus the optimal configuration is the one with the ∇ε term and 20 ≤nn ≤ 32.

5 VA L I DAT I O N O F TH E A DA P T I V ETreePM C O D E

Cosmological N-body simulations lack the equivalent of equilib-rium DFs for haloes, e.g. Plummer halo that may be used to vali-date a new code. Use of such equilibrium distributions allows one toquantify errors in a clean manner. However, given that the formalismwe use is the same as that presented by Price & Monaghan (2007),and that their implementation works well with such tests gives usconfidence in the formalism. In the following discussion, we willtest the ATreePM code in a variety of ways and look for numer-ical convergence. We compare the performance of the ATreePMwith the fixed resolution TreePM. We also study the role of the∇ε term and check if dropping this term leads to any degradation inperformance of the code. This last point is important as most AMRcodes do not have an equivalent term in the EOM.

5.1 Two-body collisions: plane wave collapse

We start by checking whether the adaptive force softening sup-presses two-body collisions in the ATreePM code. We repeat a test



recommended by Melott et al. (1997) where they study the collapseof an oblique plane wave. In this test, the collapse should not leadto any transverse motions if the evolution is collisionless. Melottet al. (1997) had shown that codes where the force softening lengthis much smaller than the interparticle separation are collisional andlead to generation of significant transverse motions. Some authors(Heitmann et al. 2005) have stated that the failure to ensure planarsymmetry may not have a one-to-one correspondence with colli-sionality. However, the test is nevertheless an important one and wepresent the results with the ATreePM code here.

We use exactly the same initial perturbations as used by Melottet al. (1997). The simulations are done with 643 particles and a 643

grid. We choose r s = 1 grid length. Other simulation parametersare as described in Section 3.1 on the TreePM code. The output isstudied at a = 3, following Melott et al. (1997).

We first conduct the test with the fixed resolution TreePM code.Top row of Fig. 3 shows the phase portrait along the direction ofcollapse for different choices of the force softening length. Thesoftening length varies between 0.1 ≤ ε ≤ 0.5 in units of the meaninterparticle separation. We see that the phase portrait in the mul-tistream region is heavily distorted for the smallest force softeninglength but is correct for the largest force softening length used here.This reinforces the conclusions of Melott et al. (1997) that using aforce softening length that is much smaller that the mean interparti-cle separation leads to two-body collisions. This point is presentedagain in the lower row of Fig. 3 where we show a scatter plot of mod-ulus of the transverse (to the direction of collapse) velocities andradial velocities. This is shown for a random subset of all particles.We see that transverse motions are significant for the simulationwith the smallest force softening length, but are under control forthe largest force softening length. Thick line in the lower panelsconnects the average modulus of the transverse velocities in bins of

magnitude of radial velocity. The visual impression gathered fromthe scatter plot is reinforced in that with decreasing force softeninglength, we get larger transverse motions.

Fig. 4 presents results of simulations with the same initial condi-tions carried out with the ATreePM code. We show the results forATreePM without the ∇ε term, nn = 16 (left-hand column); withthe ∇ε term, nn = 16 (middle column) and with the ∇ε term, nn =32 (right-hand column). In general we do not recommend use of nn

= 16 due to reasons discussed in the preceding subsection on errors,discussion in the following subsections, and in Price & Monaghan(2007), but we nevertheless use it in order to look for early signs oftwo-body collisions in a simulation with a relatively small numberof particles. The phase portrait for all the three ATreePM runs is afaithful representation of the expectations. The transverse motionsare suppressed strongly, almost to the same level as the TreePMsimulation with a force softening of ε = 0.5. One of the reasons forthis is that the highest overdensities reached in this experiment donot lead to a considerable reduction in the force softening length.The other reason is that the adaptive nature of the code leads to pres-ence of a sufficient number of neighbours within a force softeninglength and hence the anisotropy in the transverse force is reducedsubstantially.

Thick lines in the lower panels show the modulus of the averagetransverse velocities in a few bins of the modulus of the longitudi-nal velocity. We see that for TreePM, the magnitude of transversemotions drops rapidly as we increase the force softening length. Wealso note that for the ATreePM, the magnitude of transverse mo-tions is similar to that seen with the TreePM when ε for the TreePMcoincides with the εmax for the ATreePM.

We conclude the discussion of this test by noting that our re-quirement of ε � rij , where rij is the local interparticle separation,ensures collisionless behaviour in evolution of the system.

Figure 3. Top row: phase plots of the normal (to one of the planes of collapse) component of the velocity versus normal component of particle displacementfor TreePM with ε = rs/10, rs/4, rs/2 (columns 1–3, respectively). Bottom row: transverse component of velocity versus normal component of velocity alongone of the planes of collapse for the above runs. The scatter plot shows this for a random subset of particles, whereas the line shows the average value in a fewbins.



Figure 4. Top row: phase plots of the normal component of particle displacement versus normal component of the velocity for ATreePM with εi ≤ rs/2 fornn = 16 without ∇ε, nn = 16 with ∇ε and nn = 32 with ∇ε (columns 1–3, respectively). Bottom row: normal component of velocity versus transversecomponent of velocity along one of the planes of collapse for the above runs. The scatter plot shows this for a random subset of particles, whereas the lineshows the average value in a few bins.

5.2 Convergence with nn and relevance of the ∇ε term

For the following discussions we run a power-law model with indexn = −1.0. The spectrum is normalized at an epoch when the scaleof non-linearity rnl = 6.0 in grid units. The scale of non-linearity isdefined as the scale at which the linearly extrapolated mass variance,defined using a top hat filter, is unity σ (rnl, z) = 1. The simulationsare done with 1283 particles and a 1283 grid. We choose r s = 1 gridlength. Other simulation parameters are as described in Section 3.1on the TreePM code. We assume that the Einstein–de Sitter modeldescribes the background universe. In this case self-similar evolu-tion of quantities for power-law models provides an additional testfor simulation results.

5.2.1 Clustering properties

We compute the volume averaged two-point correlation function ξ :

ξ (r) = 3

r3

∫ r

0ξ (x)x2 dx, (26)

ξ is the two-point correlation function (Peebles 1980). To computeξ from simulation output, we take 15 independent random subsetsof 105 particles each. We then estimate the average value of ξ overthese subsets. The maximum and the minimum values of ξ in thesesubsets are our estimate of the errors. Fig. 5 shows ξ at an epochwhen rnl = 6.0. TreePM (left-hand column) with ε = 0.025, 0.05,0.1, 0.2, 0.5 are shown in black, red, blue, ochre and magenta,respectively. ATreePM with ∇ε (middle column) and ATreePMwithout ∇ε (right-hand column) with nn = 16, 32, 48 are shown inblue, red, black lines, respectively. The lower row is a zoom-in of ξ

so as to highlight the differences at small scales, due to the variationin ε and nn in different runs.

One can show that we require a minimum of 12 neighbours tosolve equation (20). The ∇ε term is important in overdense regionswhere εi � εmax, and is a rapidly varying function of density, onetherefore requires a reasonably large nn to compute it accurately.Use of a small nn leads to a noisy estimate of ∇ε and Fig. 5 illus-trates this point.5 With nn = 16, ξ deviates significantly from ourexpectations. Disagreement is worse for ATreePM with ∇ε, as thisis the case where a poor estimate of the extra term leads to largererrors. If we use a larger nn, we expect to see convergence at somepoint. Curves for nn = 32 and 48 agree with each other (within errorbars) for ATreePM with ∇ε indicating that evaluation of the extraterm is stable. It also indicates that this extra term compensates forthe use of a larger number of particles, otherwise typical softeninglength is expected to be larger for higher nn and this should lead tolowering of the clustering amplitude at small scales. This is veryclearly illustrated for TreePM (left-hand column in Fig. 5) whereincreasing ε reduces ξ monotonically. We also see this effect forATreePM without the ∇ε term where the amplitude of clusteringis much smaller for nn = 48 as compared to nn = 32 and the dif-ference between the two is more than twice that seen for ATreePMwith the ∇ε term. Such a behaviour is expected, and was also seenby Price & Monaghan (2007). They noted that the increase in forcesoftening length with nn leads to a bias in the force at small scales.The ∇ε term corrects for this and biasing of force is less important.Since the distribution of particles gets more strongly clustered withtime, we expect oversoftening of the force field to degrade furtherevolution for the TreePM and the ATreePM without ∇ε, beyond acertain epoch.

5 Also see Fig. 2.



Figure 5. Top row: plots the volume averaged two-point correlation function ξ as a function of scale at rnl = 6.0 for TreePM (left) and ATreePM with ∇ε

(middle) and ATreePM without ∇ε (right). The five curves (black, red, blue, ochre, mauve) for TreePM are for runs with softening lengths of ε = rs/40, rs/20,rs/10, rs/5, rs/2. For the ATreePM runs the three curves (black, red, blue) are for nn = 16, 32, 48. Bottom row: zoomed in plots of top row. Grey surface inthe ATreePM plots is ξ bounded by error bars for TreePM with ε = rs/40.

Figure 6. Tests self-similar evolution of the scaled two-point correlation function ξ (r/rnl) by plotting it at the epochs when rnl = 2.0, 4.0 and 6.0 (black, redand blue lines) for TreePM (left), ATreePM with ∇ε (middle) and ATreePM without ∇ε.

We have plotted the scaled two-point volume averaged correla-tion ξ (r/rnl) at small scales in Fig. 6 for TreePM with ε = 0.025(left-hand panel), ATreePM with the ∇ε term (middle panel) andATreePM without the ∇ε term (right-hand panel). We used simu-lations with nn = 32 for both the ATreePM runs shown here. ξ hasbeen computed at epochs when the scale of non-linearity rnl = 2, 4and 6 (black, red and blue lines, respectively). Since the only scalein power-law models is the scale of non-linearity rnl, introduced bygravity, one expects ξ to evolve in a self-similar manner. The scaleof self-similarity, rss, for a given epoch is determined by finding thescale at which ξ matches with the ξ of the earlier epoch within theerror bars computed in the manner described above.

We illustrate numerical convergence properties of the three codesin Table 1. Here we list the scale rcon and rss at different epochs.For ATreePM rcon is the scale beyond which ξ with the given nn

matches with a reference ξ with nn = 48. For TreePM this is the

scale where ξ with a given force softening length matches with thereference ξ with ε = 0.025.

We find that both variants of ATreePM converge with time, theconvergence being more rapid for ATreePM with ∇ε as compared toATreePM without ∇ε. The scale of convergence rcon is also smallerfor ATreePM with ∇ε. On the other hand, self-similar behaviour isdegraded with evolution for ATreePM without ∇ε as rss increaseswith time, whereas for ATreePM with the ∇ε term the evolution isself-similar over a larger range of scales.

Thus we may conclude that the ATreePM with the ∇ε has well-defined numerical convergence as we vary nn, and the evolution ofpower-law models for this code is self-similar over a wide range ofscales. This sets it apart from the ATreePM without the ∇ε term,where the convergence with nn is not well defined and the evolutionof a power-law model is self-similar over a smaller range of scales.Better match with the fixed resolution TreePM code is an added



Table 1. The variation of rcon and rss with time for TreePM and the two variants ofATreePM as a function of softening.

Runs rcon rcon rcon rss rss

rnl = 6.0 rnl = 4.0 rnl = 2.0 rnl = 6.0 rnl = 4.0

TreePM (ε = rs/2) 1.2 0.96 0.61 1.1 0.67TreePM (ε = rs/5) 0.61 0.44 0.36 0.42 0.15TreePM (ε = rs/10) 0.25 0.20 0.10 0.18 0.16TreePM (ε = rs/20) 0.10 0.10 0.10 0.10 0.17TreePM (ε = rs/40) – – – 0.10 0.18ATreePM (∇ε, nn = 32) 0.10 0.15 0.30 0.27 0.31ATreePM (∇ε, nn = 48) – – – 0.42 0.47ATreePM (nn = 32) 0.27 0.28 0.38 0.54 0.42ATreePM (nn = 48) – – – 0.59 0.50

positive feature for the code with the ∇ε term. This raises obviousquestions about AMR codes, where no such term is taken intoaccount as we increase the resolution of the code. We comment onthis issue in Section 6.

We briefly comment on the convergence and self-similar evolu-tion in the fixed resolution TreePM code. As seen in Table 1, wefind that at early times the scale above which we have self-similarevolution is almost the same for ε ≤ 0.2. This may indicate dis-creteness noise, or it may be due to two-body collisions duringearly collapse (Melott et al. 1997; Splinter et al. 1998; Romeo et al.2008; Joyce, Marcos & Baertschiger 2009). At late times, the scaleabove which evolution is self-similar is r ss ∼ 2ε. This indicates thatany transients introduced at early times by discreteness etc. havebeen washed out by the transfer of power from large scales to smallscales (Little, Weinberg & Park 1991; Evrard & Crone 1992; Bagla& Padmanabhan 1997b; Bagla & Prasad 2009). Thus one gets animproved self-similar evolution at late times in the fixed resolutionTreePM with the use of a smaller softening length.

On the other hand, we see that for TreePM, rcon increases withtime. As defined above, this is the scale at which ξ obtained with agiven value of ε matches with the value obtained using ε = 0.025.We find that at the last epoch rcon ≥ 2ε, whereas at early timesthe limit is rcon ≥ ε. This trend is contrary to that seen with theATreePM for which rcon decreases with time, the rate at which itcomes down being faster for the ATreePM with the ∇ε. We wouldlike to point out that we do not probe ξ for scales r < 0.1 as thenumber of pairs with a smaller separation is often too small to getreliable estimates of ξ .

Another remarkable fact is that for the TreePM code, r ss ≤ rcon.Thus we have the desired behaviour in terms of evolution eventhough we are yet to achieve numerical convergence at the relevantscales. Such a problem does not exist for the ATreePM code.

Lastly, as we show below, the ATreePM code is very good at re-solving highly overdense regions very well. This would appear to bein contradiction with the slightly lower two-point correlation func-tion. The reason for the slightly lower correlation function is thatthe ATreePM code does not resolve small haloes, in particular thosewith fewer than nn particles (see the discussion of mass functionof collapsed haloes). Indeed, the number density of small haloesis severely underestimated in the ATreePM simulations and we be-lieve that this is the main reason for a weaker two-point correlationfunction.

5.2.2 Collapsed haloes

We now look at another global indicator, namely the mass functionof collapsed haloes, to compare the performance of the TreePM andthe ATreePM codes. The number density of haloes N (M) dM in themass range (M , M + dM) is plotted in Fig. 7. Haloes were identifiedusing the friends-of-friends algorithm with a linking length ll = 0.1in grid units and only haloes with at least eight particles wereconsidered.

For TreePM we see that as we increase the force softening lengththere is a clear change in the mass function. The mass functionconverges for all masses with ε ≤ 0.2. However, the mass functionfor ε = 0.5 shows fewer haloes of up to about 102 particles. Thus

Figure 7. Mass function for TreePM (left), ATreePM with ∇ε (middle) and ATreePM without ∇ε (right). For TreePM ε = rs/40, rs/10, rs/2 are drawn in black,red, blue lines, respectively. nn = 32 and 48 are drawn in black and red lines for ATreePM. The arrows represent nn in mass units. The black dot–dashed curveon every plot is the Press–Schechter mass function, shifted vertically by multiplying by a factor of 1.5. The black dashed line in ATreePM is for TreePM withε = rs/40.



the effect of a large softening length is seen in mass function upto fairly large mass scales. This qualitative behaviour is insensitiveto the choice of linking length, though the scale of convergence issmaller if we choose a larger linking length.

The ATreePM does not sample the haloes with fewer than nn

members properly. This is expected as such overdensities are notsufficient to reduce the softening length from εmax. At larger scaleswe have a convergence for nn = 32 and 48, where the mass functionagrees with that for TreePM with ε = 0.025. Mass function for nn =16 deviates significantly from the other two for ATreePM, deviationbeing larger for the ATreePM with the ∇ε term. We believe that thisis due to the noisy estimation of the force softening length and the∇ε term, and resulting errors in force.

As a reference the Press–Schechter mass function is also plottedin all three panels (black dot–dashed line). However, it has beenshifted up vertically by multiplying N (M) dM by a factor of 1.5 forease of comparison.

Distribution of particles in a simulation is a useful representationfor comparing large-scale features. The distribution of particles in athin slice is shown in Fig. 8. We have shown the distribution for foursimulations: TreePM with ε = 0.025 (top left), TreePM with ε =

0.5 (top right), ATreePM with the ∇ε term and nn = 32 (lower left)and ATreePM without the ∇ε term and nn = 32 (lower right). Thedistribution of particles is for the last epoch, corresponding to rnl =6 grid lengths. This slice contains the largest halo in our simulationon the top left-hand corner. We note that the large-scale distributionis the same in all cases, indicating that the ATreePM is not changinganything at large scales.

We zoom in on the region around the largest halo in Fig. 9, wherethe distribution of particles is shown from the set of simulations usedin Fig. 8. Distribution of mass at scales larger than a grid length isthe same in all simulations but at small scales we begin to seesome differences between the distributions of particles in differentsimulations. TreePM with ε = 0.5 differs most from the other three,in that it does not resolve small-scale structures. This happens due tothe relatively large force softening length that inhibits collapse if theexpected size of the collapsed halo is smaller than ε. The notabledifferences between the TreePM (ε = 0.025) and the ATreePMslices are as follows.

(i) Small haloes in the region away from the large haloes are morecompact for TreePM than for ATreePM. This is perhaps caused by

Figure 8. Slice from simulation where the most massive halo resides (see top left-hand corner). Top panel is for TreePM with ε = rs/40 (left) and ATreePMwith ε = rs/2. Bottom panel is for ATreePM with ∇ε (left) and ATreePM without ∇ε (right). nn = 32 was taken for ATreePM.



Figure 9. This is a zoom in of Fig. 8 and shows the region of simulation where the most massive halo resides (see top left-hand corner of Fig. 8). Top panel isfor TreePM with ε = rs/40 (left) and ATreePM with ε = rs/2. Bottom panel is for ATreePM with ∇ε (left) and ATreePM without ∇ε (right). nn = 32 wastaken for ATreePM.

the ATreePM not having a constant ε and it is likely that in theseclumps the value of ε is larger than 0.025 that is used in the TreePM.

(ii) Location of substructure in the central halo differs somewhatbetween the different simulations.

Fig. 10 shows the inner regions of the halo seen in Fig. 9. Weshow the inner parts of the halo in the four cases [TreePM (ε =0.025), ATreePM with the ∇ε term (nn = 32), ATreePM withoutthe ∇ε term (nn = 32)]. The top row shows all the particles in theinner parts of the halo. The second row shows all the particles withan SPH overdensity estimate (computed with nn = 32 and usingthe kernel specified in equation 5) of greater than 5 × 104, thethird row shows the particles with an overdensity of greater than105 and the last row shows particles with an overdensity larger than2 × 105. We see that the ATreePM with the ∇ε term shows themost pronounced central part of the halo in the top panel, and thisimpression gains strength as we move to lower panels. Table 2 givesus the number of particles in the various panels of Fig. 10. TreePMwith ε = 0.025 manages to trace some of the highly overdensesubstructure seen in the ATreePM simulations, though with a muchsmaller number of particles in these clumps. The ATreePM with the

∇ε term does better than the one without, particularly at the highestoverdensities used here. Indeed ATreePM with the ∇ε term retainsnearly seven times as many particles in the halo core as comparedto TreePM and ATreePM without the ∇ε term. The same thing isalso reflected in Fig. 5 where the ATreePM without the ∇ε termis seen to underestimate clustering at small scales. It is noteworthythat the size of highly overdense structures in the core of this halois much bigger than the force softening length for the TreePM.

It is notable that the ATreePM is able to resolve highly over-dense regions while taking much less time than the fixed resolutionTreePM. Thus we have the added performance at the cost of fewerresources.

5.2.3 Dynamics within collapsed haloes

The EOM for the adaptive code has the additional ∇ε term, and itis important to test whether this term leads to a change in dynamicsin highly overdense regions. We test this by studying the dynamicsin central cores of the largest haloes in the simulation. We studythe ratio 〈2T 〉/〈U〉 at a scale corresponding to r200/5, with theaveraging done around the centre of mass of the halo. We chose



Figure 10. The largest halo at rnl = 6.0 for TreePM with ε = rs/40 (left-hand column), ATreePM with ∇ε and nn = 48 (middle column) and ATreePMwithout ∇ε and nn = 32 (right-hand column). The first row contains all the particles in the halo. The second, third and forth rows represent particles in thehalo which have a density greater than 5 × 104, 105 and 2 × 105, respectively.

Table 2. The number of particles npart above a density threshold (asin Fig. 10) retained in the core of the most massive halo.

Runs npart npart npart

ρ ≥ 5 × 104 ρ ≥ 105 ρ ≥ 2 × 105

TreePM (ε = rs/40) 320 106 18ATreePM (∇ε, nn = 32) 568 297 125ATreePM (nn = 32) 374 169 18

the reference scale in this manner as the density profiles for haloeswith different softening length differ considerably and using a high-density contour to define the radius leads to very different physicalscales. Fig. 11 shows 〈2T 〉/〈U〉 as a function of softening length ε

for the fixed resolution TreePM simulations. This has been done forthe five largest haloes in the simulation. We see that as we go to alarger softening length, the ratio becomes larger than unity, indeedfor ε = 0.5 the ratio is closer to two for one of the haloes. Thisis likely to happen if the size of the overdense core is comparable



Figure 11. Ratio of two times the kinetic energy and the potential energy,2T /U , for the five largest haloes as a function of softening length for TreePM(black lines). Open squares and filled triangles are for ATreePM with andwithout ∇ε, respectively.

to the softening length, and indeed this is the case for some of thehaloes used for this plot. The same plot shows the ratio as seen inthe adaptive code (with the ∇ε term) with empty squares, and asfilled triangles for the adaptive code without the ∇ε term. Thesecircles are plotted at small values of ε, not used for TreePM runs.The values of ε used for positioning these symbols in the plot hasno relevance to the simulation, and this has been done purely forthe purpose of plotting the values. We see that the values for theratio in both the adaptive codes correspond closely to the values ofthe ratio seen in fixed resolution TreePM simulations with a smallε. Thus we may conclude that the additional term in the adaptivecode is not leading to any significant changes in the dynamics, ascompared to the fixed resolution codes.

6 D ISCUSSION

In this paper we have introduced the first cosmological N-body codethat has a continuously varying force resolution. This is based on awell-defined formalism that ensures energy and momentum conser-vation. An important aspect of this formalism is that it modifies theEOM at scales below the force softening length. Our implementa-tion of the formalism uses methods developed for SPH codes, thisis required as we need to assign quantities like density to particles.We have described our implementation of the code in detail here.We have given a brief summary of errors in force for the ATreePMcode with the new parameter nn. Given that this code is based onthe TreePM code, it inherits the errors in force with respect to allthe other parameters. As we have seen in earlier work (Bagla 2002;Bagla & Ray 2003; Khandai & Bagla 2009), errors in force canbe minimized to a fairly low level for the TreePM with a judiciouschoice of parameters.

We find that the time taken by the ATreePM (for nn = 32) iscomparable to that taken by the fixed resolution TreePM code if thelatter uses a force softening length that is about 1/20 of the meaninterparticle separation.

One of the key reasons for considering adaptive resolution is toavoid two-body collisions. Cosmological simulations require col-lisionless evolution. We test the ATreePM by simulating collapse

of an oblique plane wave. We find that unlike the fixed resolutionTreePM code that leads to large transverse motions, a clear sign oftwo-body collisions, the ATreePM code does not have any signifi-cant transverse motions (see Figs 3 and 4).

We have compared the performance of the ATreePM with that ofthe fixed resolution TreePM code for power-law initial conditions.We used a power-law model and the Einstein–de Sitter backgroundas it allows us to test codes by requiring self-similar evolution ofthe correlation function. We find that for nn = 32, the resultingdensity distribution in the ATreePM is comparable to that seen inthe highest resolution TreePM in many respects. The correlationfunction for the two match at all scales and the mass function ofcollapsed haloes matches for haloes with more than nn particles. TheATreePM with the ∇ε term in the EOM does much better than theTreePM as well as the ATreePM without the extra term in resolvinghighly overdense cores of massive haloes. It is noteworthy that theadaptive code takes less time than the TreePM it has been comparedwith (see Fig. 1).

We have tested the codes by looking for convergence in results aswe modify the key parameters that describe the force resolution. Wefind that the fixed resolution TreePM code converges slower thanexpected from, for example, self-similar evolution of clustering.The ATreePM code with the ∇ε term converges fairly quickly andoffers an effective dynamical range that is slightly smaller than thefixed resolution TreePM that takes almost the same time to run.

Given our analysis of errors, and the comparison of the perfor-mance of the ATreePM code with and without the extra term in theEOM, the natural conclusion is that we require the extra term inorder to obtain low errors and numerical convergence. This raisesthe obvious question about the AMR codes where no such extraterm is used. Further, in most AMR codes, resolution is increasedwhen there are of order 10 particles in the lower resolution ele-ments. We find that the errors are minimized when this number isaround 20 even when the extra term is not taken into account. It isnot clear how serious these issues are, given that AMR codes havebeen developed and tested in a variety of ways over the last threedecades, but it is a concern.6

We have established in this paper that an adaptive resolution codefor evolution of perturbations in collisionless dark matter can givereliable results for a range of indicators from clustering properties tomass function of collapsed haloes and even get the internal dynamicsof collapsed haloes tight. Further, we show that such a code isefficient in that it is faster than fixed resolution codes that give uscomparable force resolution in highly overdense regions. Such acode is very useful as it allows us to probe clustering at small scalesin a reliable manner. Studies of box-size effects have shown thatlarge simulation boxes are required in order to limit the effect ofperturbations at larger scales that are not taken into account (Bagla& Prasad 2006; Bagla et al. 2009). In such a situation an adaptivecode provides us with a reasonable range of scales over which theresults can be trusted. We expect to use this code to address severalissues related to gravitational clustering in an expanding universe.We also plan to revisit issues related to density profiles of collapsedhaloes.

6 One aspect where AMR codes do relatively poorly is in getting the halomass function at the low-mass end (O’Shea et al. 2005; Heitmann et al.2008). As one can see in figures in the papers cited here, the shortfall isoften spread over a decade in mass of haloes. We do not see any gradualdecline in the number density of haloes in ATreePM up to the cut-off set bynn.



Given the conclusions listed above, we feel that the ATreePMmethod represents an exciting development where we can set asideworries about the impact of collisionality. The relative speed of theadaptive code also makes this a more pragmatic option.

AC K N OW L E D G M E N T S

The authors are indebted to Daniel J. Price for clarifying sev-eral issues regarding their implementation of the adaptive code.The authors thank Martin White, Volker Springel and AlessandroRomeo for useful discussions and comments. Numerical experi-ments for this study were carried out at cluster computing facilityin the Harish-Chandra Research Institute (http://cluster.hri.res.in).This research has made use of NASA’s Astrophysics Data System.

REFERENCES

Athanassoula E., Fady E., Lambert J. C., Bosma A., 2000, MNRAS, 314,475

Bagla J. S., 2002, JA&A, 23, 185Bagla J. S., 2005, Curr. Sci., 88, 1088Bagla J. S., Padmanabhan T., 1994, MNRAS, 266, 227Bagla J. S., Padmanabhan T., 1997a, Pramana, 49, 161Bagla J. S., Padmanabhan T., 1997b, MNRAS, 286, 1023Bagla J. S., Prasad J., 2006, MNRAS, 370, 993Bagla J. S., Prasad J., 2009, MNRAS, 393, 607Bagla J. S., Ray S., 2003, New Astron., 8, 665Bagla J. S., Ray S., 2005, MNRAS, 358, 1076Bagla J. S., Prasad J., Khandai N., 2009, MNRAS, 395, 918Barnes J. E., 1990, J. Comput. Phys., 87, 161Barnes J., Hut P., 1986, Nat, 324, 446Bernardeau F., Colombi S., Gaztanaga E., Scoccimarro R., 2002, Phys. Rep.,

367, 1Bertschinger E., 1998, ARA&A, 36, 599Binney J., 2004, MNRAS, 350, 939Binney J., Knebe A., 2002, MNRAS, 333, 378Bode P., Ostriker J. P., 2003, ApJS, 145, 1Bode P., Ostriker J. P., Xu G., 2000, ApJS, 128, 561Bouchet F. R., Hernquist L., 1988, ApJS, 68, 521Bouchet F. R., Kandrup H. E., 1985, ApJ, 299, 1Bouchet F. R., Adam J.-C., Pellat R., 1985, A&A, 144, 413Brainerd T. G., Scherrer R. J., Villumsen J. V., 1993, ApJ, 418, 570Brieu P. P., Evrard A. E., 2000, New Astron., 5, 163Brieu P. P., Summers F. J., Ostriker J. P., 1995, ApJ, 453, 566Couchman H. M. P., 1991, ApJ, 368, L23Couchman H. M. P., Thomas P. A., Pearce F. R., 1995, ApJ, 452, 797Davis M., Peebles P. J. E., 1977, ApJS, 34, 425Dehnen W., 2000, ApJ, 536, L39Dehnen W., 2001, MNRAS, 324, 273Dehnen W., 2002, J. Comput. Phys., 179, 27Diemand J., Moore B., Stadel J., Kazantzidis S., 2004, MNRAS, 348, 977Dolag K., Borgani S., Schindler S., Diaferio A., Bykov A. M., 2008, Space

Sci. Rev., 134, 229Dubinski J., 1996, New Astron., 1, 133Dubinski J., Kim J., Park C., Humble R., 2004, New Astron., 9, 111Ebisuzaki T., Makino J., Fukushige T., Taiji M., Sugimoto D., Ito T.,

Okumura S. K., 1993, PASJ, 45, 269Efstathiou G., Davis M., White S. D. M., Frenk C. S., 1985, ApJS, 57, 241Efstathiou G., Frenk C. S., White S. D. M., Davis M., 1988, MNRAS, 235,

715El-Zant A. A., 2006, MNRAS, 370, 1247Evrard A. E., 1988, MNRAS, 235, 911Evrard A. E., Crone M. M., 1992, ApJ, 394, L1Gingold R. A., Monaghan J. J., 1977, MNRAS, 181, 375Gingold R. A., Monaghan J. J., 1982, J. Comput. Phys., 46, 429Greengard L., Rokhlin V., 1987, J. Comput. Phys., 73, 325

Gurbatov S. N., Saichev A. I., Shandarin S. F., 1989, MNRAS, 236,385

Hamilton A. J. S., Kumar P., Lu E., Matthews A., 1991, ApJ, 374, L1Heitmann K., Ricker P. M., Warren M. S., Habib S., 2005, ApJS, 160, 28Heitmann K. et al., 2008, Comput. Sci. Disc., 1, 015003Hernquist L., 1990, J. Comput. Phys., 87, 137Hernquist L., Barnes J. E., 1990, ApJ, 349, 562Hernquist L., Katz N., 1989, ApJS, 70, 419Hernquist L., Bouchet F. R., Suto Y., 1991, ApJS, 75, 231Hockney R. W., Eastwood J. W., 1988, Computer Simulation Using Particles.

McGraw-Hill, New YorkHui L., Bertschinger E., 1996, ApJ, 471, 1Jain B., Mo H. J., White S. D. M., 1995, MNRAS, 276, L25Jernigan J. G., Porter D. H., 1989, ApJS, 71, 871Joyce M., Marcos B., Baertschiger T., 2009, MNRAS, 394, 751Kanekar N., 2000, ApJ, 531, 17Kawai A., Makino J., 2001, ApJ, 550, L143Khandai N., Bagla J. S., 2009, Res. Astron. Astrophys., in press

(arXiv:0802.3215)Klypin A. A., Shandarin S. F., 1983, MNRAS, 204, 891Knebe A., Green A., Binney J., 2001, MNRAS, 325, 845Kravtsov A. V., Klypin A. A., Khokhlov A. M., 1997, ApJS, 111, 73Kuhlman B., Melott A. L., Shandarin S. F., 1996, ApJ, 470, L41Little B., Weinberg D. H., Park C., 1991, MNRAS, 253, 295Lucy L. B., 1977, AJ, 82, 1013Ma C.-P., 1998, ApJ, 508, L5Macfarland T., Couchman H. M. P., Pearce F. R., Pichlmeier J., 1998, New

Astron., 3, 687Makino J., 1990, J. Comput. Phys., 87, 148Makino J., 1991, PASJ, 43, 621Makino J., 2002, New Astron., 7, 373Makino J., 2004, PASJ, 56, 521Makino J., Fukushige T., Koga M., Namura K., 2003, PASJ, 55, 1163Matarrese S., Lucchin F., Moscardini L., Saez D., 1992, MNRAS, 259,

437Melott A. L., Shandarin S. F., Splinter R. J., Suto Y., 1997, ApJ, 479, L79Merritt D., 1996, AJ, 111, 2462Merz H., Pen U.-L., Trac H., 2005, New Astron., 10, 393Miller R. H., 1983, ApJ, 270, 390Monaghan J. J., 1992, ARA&A, 30, 543Monaghan J. J., Lattanzio J. C., 1985, A&A, 149, 135Nityananda R., Padmanabhan T., 1994, MNRAS, 271, 976O’Shea B. W., Nagamine K., Springel V., Hernquist L., Norman M. L.,

2005, ApJS, 160, 1Padmanabhan T., 1996, MNRAS, 278, L29Padmanabhan T., 2002, Theoretical Astrophysics, Vol. III, Galaxies and

Cosmology. Cambridge Univ. Press, CambridgePadmanabhan T., Cen R., Ostriker J. P., Summers F. J., 1996, ApJ, 466, 604Peacock J. A., 1999, Cosmological Physics. Cambridge Univ. Press,

CambridgePeacock J. A., Dodds S. J., 1994, MNRAS, 267, 1020Peacock J. A., Dodds S. J., 1996, MNRAS, 280, L19Peebles P. J. E., 1974, A&A, 32, 391Peebles P. J. E., 1980, The Large-Scale Structure of the Universe. Princeton

Univ. Press, Princeton, NJPercival W. J. et al., 2007, ApJ, 657, 645Power C., Knebe A., 2006, MNRAS, 370, 691Prasad J., 2007, JA&A, 28, 117Price D. J., Monaghan J. J., 2007, MNRAS, 374, 1347Ray S., Bagla J. S., 2004, preprint (astro-ph/0405220)Romeo A. B., Agertz O., Moore B., Stadel J., 2008, ApJ, 686, 1Sahni V., Coles P., 1995, Phys. Rep., 262, 1Salmon J. K., Warren M. S., 1994, J. Comput. Phys., 111, 136Smith R. E. et al., 2003, MNRAS, 341, 1311Spergel D. N. et al., 2007, ApJS, 170, 377Splinter R. J., Melott A. L., Shandarin S. F., Suto Y., 1998, ApJ, 497, 38Springel V., 2005, MNRAS, 364, 1105Springel V., Hernquist L., 2002, MNRAS, 333, 649



Springel V., Yoshida N., White S. D. M., 2001, New Astron., 6, 79Springel V. et al., 2005, Nat, 435, 629Suisalu I., Saar E., 1995, MNRAS, 274, 287Thacker R. J., Couchman H. M. P., 2006, Comput. Phys. Commun., 174,

540Theuns T., 1994, Comput. Phys. Commun., 78, 238

Thiebaut J., Pichon C., Sousbie T., Prunet S., Pogosyan D., 2008, MNRAS,387, 397

Wadsley J. W., Stadel J., Quinn T., 2004, New Astron., 9, 137Xu G., 1995, ApJS, 98, 355Yoshikawa K., Fukushige T., 2005, PASJ, 57, 849Zel’Dovich Y. B., 1970, A&A, 5, 84

APPENDIX A : EXPRESSIONS FOR CUBIC SPLI NE SOFTENED POTENTI AL AND FORCE K ERNELS

One can integrate the Poisson equation for a given kernel to obtain the softened two-body potential kernel. For the case of the cubic splinekernel we have the expression for the two-body potential kernel φ(u, ε) and the regular two-body force f (u, ε) = −∇u φ:

φ(u, ε) =

⎧⎪⎪⎪⎨⎪⎪⎪⎩16ε

[13

(u

ε

)2 − 35

(u

ε

)4 + 25

(u

ε

)5] − 145ε

, 0 ≤ u

ε< 0.5,

1ε

[115

(ε

u

) + 323

(u

ε

)2 − 161

(u

ε

)3 + 485

(u

ε

)4 − 3215

(u

ε

)5] − 165ε

, 0.5 ≤ u

ε< 1.0,

− 1u, 1.0 ≤ u

ε,

(A1)

f (u, ε) =

⎧⎪⎪⎪⎨⎪⎪⎪⎩− 32u

ε3

[13 − 6

5

(u

ε

)2 + (u

ε

)3], 0 ≤ u

ε< 0.5,

− uε3

[− 1

15

(ε

u

)3 + 643 − 48u

ε+ 192

5

(u

ε

)2 − 323

(u

ε

)3], 0.5 ≤ u

ε< 1.0,

− uu3 , 1.0 ≤ u

ε.

(A2)

For variable or local smoothing length εi ≡ ε(r i) there will be an additional term given in equations (10)–(12) for which one needs to compute(∂W/∂ε), (∂W/∂u) and (∂φ/∂ε). For the cubic spline kernel these expressions are

∂W

∂ε= 8

πε4

⎧⎪⎪⎨⎪⎪⎩3

[−1 + 10

(u

ε

)2 − 12(

u

ε

)3], 0 ≤ u

ε< 0.5,

6[−1 + 4

(u

ε

) − 5(

u

ε

)2 + 2(

u

ε

)3], 0.5 ≤ u

ε< 1.0,

0, 1.0 ≤ u

ε,

(A3)

∂W

∂u= 48

πε4

uu

⎧⎪⎨⎪⎩−2

(u

ε

) + 3(

u

ε

)2, 0 ≤ u

ε< 0.5,

−1 + 2(

u

ε

) − (u

ε

)2, 0.5 ≤ u

ε< 1.0,

0, 1.0 ≤ u

ε,

(A4)

∂φ

∂ε=

⎧⎪⎪⎨⎪⎪⎩16ε2

[− (

u

ε

)2 + 3(

u

ε

)4 − 125

(u

ε

)5]

+ 145ε2 , 0 ≤ u

ε< 0.5,

1ε2

[−32

(u

ε

)2 + 64(

u

ε

)3 − 48(

u

ε

)4 + 645

(u

ε

)5]

+ 165ε2 , 0.5 ≤ u

ε< 1.0,

0, 1.0 ≤ u

ε.

(A5)

This paper has been typeset from a TEX/LATEX file prepared by the author.


Documents

The Adaptive TreePM: an adaptive resolution code for cosmological N-body simulations