Upload
julius-singleton
View
219
Download
0
Embed Size (px)
Citation preview
University of EdinburghJanuary 2011
DL_POLY_4
Ilian Todorov @ CCG
CSED, STFC - Daresbury Laboratory, DaresburyWarrington WA4 1EP, Cheshire, England, UK
University of EdinburghJanuary 2011
Where is Daresbury?
University of EdinburghJanuary 2011
DL_POLY Project Background
• General purpose parallel (classical) MD simulation software • It was conceived to meet the needs of CCP5 - The Computer Simulation of Condensed Phases (academic collaboration community)• Written in modularised Fortran90 (NagWare & FORCHECK compliant) with MPI2 (MPI1+MPI-I/O) fully self-contained
• 1994 – 2011: DL_POLY_2 (RD) by W. Smith & T.R. Forester
(funded for 6 years by EPSRC at DL) -> DL_POLY_CLASSIC• 2003 – 2011: DL_POLY_3 (DD) by I.T. Todorov & W. Smith
(funded for 4 years by NERC at Cambridge) -> DL_POLY_4
• Over 13,000 licences taken out since 1994 (~1500 annually)• Over 1100 registered FORUM members since 2005• Available free of charge (under licence) to University researchers (provided as code) and at cost to industry
University of EdinburghJanuary 2011
DL_POLY_DD Development Statistics
University of EdinburghJanuary 2011
DL_POLY_DD Licence Statistics
University of EdinburghJanuary 2011
DL_POLY Licence Statistics
Asia38%
EU-REST17%
North America22%
UK11%
Latin America4%
Europe-REST5%
Africa2%
Australia and New Zealand and Philipnes
1%
DL_POLY Licences2010 by Sub-Areas
Asia39%
EU-REST19%
North America15%
UK10%
Latin America8%
Europe-REST4%
Africa3%
Australia and New Zealand and Philipnes
2%
DL_POLY Licences2011 by Sub-Areas
University of EdinburghJanuary 2011
DL_POLY Licence Statistics
USA19%
China17%
UK11%
India7%
Iran5%
Germany3%
France3%
Japan3%
Russia3%
Italy2%
Canada2%
Taiwan2%
Chile2%
Mexico1%
Spain1%
Australia1%
South Korea1%
REST OF WORLD17%Other
42%
DL_POLY Licences 2010 by Countries
University of EdinburghJanuary 2011
DL_POLY Licence Statistics
P.R. China16%
USA14%
UK10%
India7%
Iran5%
France4%
Germany3%
Italy3%
Brazil3%
Japan3%
Russia3%
Spain2%
Mexico2%
Taiwan2%
Indonesia2%
Canada1%
Poland1%
Singapore1%
South Korea1%Sweden
1%
Algeria1%
Australia1%
REST OF WORLD14%
Other48%
DL_POLY Licences 2011 by Countries
University of EdinburghJanuary 2011
DL_POLY Licence Statistics
Bio-Chemsitry5%
Chemistry33%
Engineering13%
Mechanics10%Other
1%
Physics33%
Software5%
DL_POLY Licences2010 by Science Domain
Bio-Chemsitry4%
Chemistry38%
Engineering17%
Materials7%Mechanics
3%
Other3%
Physics28%
DL_POLY Licences2011 by Science Domain
University of EdinburghJanuary 2011
DL_POLY Project Current State
• October 2010: DL_POLY_3 -> DL_POLY_4 still under STFC Licence, over 1600 licences taken out since November 2010
• Rigid Body dynamics (ITT)• Parallel I/O & netCDF I/O – NAG dCSE (IJB & ITT)• CUDA+OpenMP port (as source, ICHEC) & MS Windows ports (as MSI 32- and 64-bit self-installers)• SPME processor grid freed from 2^N decomposition – NAG dCSE (IJB)
• January 2011: DL_POLY_2 -> DL_POLY_CLASSIC on a BSD type Licence (BS retired but supporting GUI and fixes)• Load Balancer development (finished 30/03/2011)• Continuous Development of the DL_FIELD field builder (pdb to DL_POLY input, Chin Yong), over 400 licences since November 2010• October 2011: new DL_POLY GUI (Bill Smith)
University of EdinburghJanuary 2011
DL_POLY Project Roadmap
• August 2011 (PRACE-2IP funding):• further CUDA porting & support by ICHEC (2 PM)• CUDA to OpenCL port by SC@WUT (5 PM)• FMM library testing as substitute of SPME electrostatics
• October 2011 (2 FTE years Software Development EPSRC funding with Kostya Trachenko)
• Two temperature thermostat methodology (TTM) for high energy events in metals (Michael Seaton)• Fragmented I/O and on-the-fly thermodynamic properties calculations for very large systems (Michael Seaton)
• November 2011 (1 FTE year dCSE funding with David Quigley):
• OpenMP within the vanilla DD/MPI framework DL_POLY_4 (Ian Bush & Asimina Maniopoulou)• Beyond 2.1 billion particles limit – long integers (Ian Bush)
University of EdinburghJanuary 2011
DL_POLY Project Roadmap II
• December 2011 (MMM@HPC - FP7-eInfrastructure funding, 3 FTE years)
• Gay-Berne particles (Laurence Ellison)
• Implementing Gentle Stochastic Thermostat
• Extension of NsT ensemble to handle liquid bio-chemical systems
• nfold and bio-chemical verification
• Modified Tersoff potential for band-gap materials
• Porting the DL_POLY_Classic Hyper-dynamics
University of EdinburghJanuary 2011
• DL_POLY_4 (version 4.02)– Dynamic Decomposition parallelisation, based on
domain decomposition but with dynamic load balancing– limits up to ≈2.1×109 atoms with inherent
parallelisation.– Full force field and molecular description with rigid
body description– Free format (flexible) reading with some fail-safe
features and basic reporting (but fully fool-proofed)• DL_POLY Classic (version 1.8)
– Replicated Data parallelisation, limits up to ≈30,000 atoms with good parallelisation up to 64 (system dependent) processors (running on any processor count)
– Full force field and molecular description– Hyper-dynamics: Temperature Accelerated Dynamics &
Biased Potential Dynamics, Solvation Dynamics – Spectral Shifts, Metadynamics, (Path Integral MD)
– Free format reading but somewhat strict
Current Versions
University of EdinburghJanuary 2011
Supported Molecular Entities
Point ionsand atoms
Polarisableions (core+shell)
Flexiblemolecules
Rigidbonds
Rigidmolecules
Flexiblylinked rigidmolecules
Rigid bondlinked rigidmolecules
University of EdinburghJanuary 2011
Force Field Definitions – I
• particle: rigid ion or atom (charged or not), a core or a shell of a polarisable ion(with or without associated degrees of freedom), a massless charged site. A particle is a countable object and has a global ID index.
• site: a particle prototype that serves to defines the chemical & physical nature (topology/connectivity/stoichiometry) of a particle (mass, charge, frozen-ness). Sites are not atoms they are prototypes!
• Intra-molecular interactions: chemical bonds, bond angles, dihedral angles, improper dihedral angles, inversions. Usually, the members in a unit do not interact via an inter-molecular term. However, this can be overridden for some interactions. These are defined by site.
• Inter-molecular interactions: van der Waals, metal (EAM, Gupta, Finnis-Sinclair, Sutton-Chen), Tersoff, three-body, four-body. Defined by species.
University of EdinburghJanuary 2011
Force Field Definitions – II
• Electrostatics: Standard Ewald*, Hautman-Klein (2D) Ewald*, SPM Ewald (3D FFTs), Force-Shifted Coulomb, Reaction Field, Fennell damped FSC+RF, Distance dependent dielectric constant, Fuchs correction for non charge neutral MD cells.
• Ion polarisation via Dynamic (Adiabatic) or Relaxed shell model.
• External fields: Electric, Magnetic, Gravitational ,Oscillating & Continuous Shear, Containing Sphere, Repulsive Wall.
• Intra-molecular like interactions: tethers, core shells units, constraint and PMF units, rigid body units. These are also defined by site.
• Potentials: parameterised analytical forms defining the interactions. These are always spherically symmetric!
• THE CHEMICAL NATURE OF PARTICLES DOES NOT CHANGE IN SPACE AND TIME!!!
University of EdinburghJanuary 2011
Force Field by Sums
i
N
1iexternal
N
ijishell-coreshell-core
N
i0tttethertether
N
idcbainversinvers
N
idcbadiheddihed
N
icbaangleangle
N
ibabondbond
N'
ji,
N'
ji,jiij
N
ijipairmetal
N'
nk,j,i,nkjibody4
N'
kj,i,kjibody3
N'
kj,i,kjiTersoff
N'
ji, ji
ji
0
N'
ji,jipairN21
rΦ|rr|,iUr,r,iU
r,r,r,r,iUr,r,r,r,iU
r,r,r,iUr,r,iU
)|rr|(ρF|)rr(|Vε
r,r,r,rUr,r,rUr,r,rU
|rr|
4π
1|)rr(|U)r,.....,r,rV(
-shellcore
-shellcore
tether
tether
invers
invers
dihed
dihed
angle
angle
bond
bond
University of EdinburghJanuary 2011
Ensembles and Algorithms
Integration:
Available as velocity Verlet (VV) or leapfrog Verlet (LFV) generating flavours of the following ensembles• NVE
• NVT (Ekin) Evans
• NVT Andersen^, Langevin^, Berendsen, Nosé-Hoover• NPT Langevin^, Berendsen, Nosé-Hoover, Martyna-Tuckerman-Klein^
• NT/NPnAT/NPnT Langevin^, Berendsen, Nosé-Hoover, Martyna-Tuckerman-Klein^
Constraints & Rigid Body Solvers: • VV dependent – RATTLE, No_Squish, QSHAKE*• LFV dependent – SHAKE, Euler-Quaternion, QSHAKE*
University of EdinburghJanuary 2011
19
AA BB
CC DD
Domain Decomposition
University of EdinburghJanuary 2011
Glo
bal fo
rce fi
eld
Glo
bal fo
rce fi
eld
PP00LocalLocalatomicatomicindicesindices
PP11LocalLocalatomicatomicindicesindices
PP22LocalLocalatomicatomicindicesindices
Pro
cess
or
Dom
ain
sPro
cess
or
Dom
ain
s
Tricky!Tricky!Molecular Molecular force fieldforce fielddefinitiondefinition
Bonded Forces within DD
University of EdinburghJanuary 2011
U. Essmann, L. Perera, M.L. Berkowtz, T. Darden, H. Lee, L.G. Pedersen, J. Chem. Phys., 103, 8577 (1995)
1. Calculate self interaction correction2. Initialise FFT routine (FFT – IJB’s DaFT: 3M2 1D FFT)3. Calculate B-spline coefficients4. Convert atomic coordinates to scaled fractional units5. Construct B-splines6. Construct partial charge array Q7. Calculate FFT of Q array8. Construct partial array G9. Calculate FFT of G array10. Calculate net Coulombic energy11. Calculate atomic forces
I.J. Bush, I.T. Todorov, W. Smith, Comp. Phys. Commun., 175, 323 (2006)
DD Scheme for long-ranged part of SPME
University of EdinburghJanuary 2011
0 200 400 600 800 1000
0
200
400
600
800
1000
max load 700'000 atoms per 1GB/CPUmax load 220'000 ions per 1GB/CPUmax load 210'000 ions per 1GB/CPU
Solid Ar (32'000 atoms per CPU) NaCl (27'000 ions per CPU) SPC Water (20'736 ions per CPU)
21 million atoms
28 million atoms
33 million atoms
Sp
ee
d G
ain
Processor Count
Performance Weak Scaling on IBM p575 2005-2011
University of EdinburghJanuary 2011
Rigid Bodies versus Constraints450,000 particles with DL_POLY_4
0
1
2
3
4
5
6
7
8
9
10
0 100 200 300 400 500 600
step
s per
sec
ond
Np
Scaling
ICE7
ICE7_CB
University of EdinburghJanuary 2011
I/O Solutions in DL_POLY_4
1. Serial read and write (sorted/unsorted) – where only a single MPI task, the master, handles it all and all the rest communicate in turn to or get broadcasted to while the master completes writing a configuration of the time evolution.
2. Parallel write via direct access or MPI-I/O (sorted/unsorted) – where ALL / SOME MPI tasks print in the same file in some orderly manner so (no overlapping occurs using Fortran direct access printing. However, it should be noted that the behaviour of this method is not defined by the Fortran standard, and in particular we have experienced problems when disk cache is not coherent with the memory).
3. Parallel read via MPI-I/O or Fortran
4. 4. Serial NetCDF read and writeSerial NetCDF read and write using NetCDF libraries using NetCDF libraries for machine-independent data formats of array-based, for machine-independent data formats of array-based, scientific data (widely used by various scientific scientific data (widely used by various scientific communities).communities).
University of EdinburghJanuary 2011
3.09 3.10 3.09 3.10Cores I/O Procs Time/s Time/s Mbyte/s Mbyte/s
32 32 143.30 1.27 0.44 49.7864 64 48.99 0.49 1.29 128.46
128 128 39.59 0.53 1.59 118.11256 128 68.08 0.43 0.93 147.71512 256 113.97 1.33 0.55 47.60
1024 256 112.79 1.20 0.56 52.472048 512 135.97 0.95 0.46 66.39
MPI-I/O Write Performance for216,000 Ions of NaCl on XT5
University of EdinburghJanuary 2011
3.10 New 3.10 NewCores I/O Procs Time/s Time/s Mbyte/s Mbyte/s
32 16 3.71 0.29 17.01 219.7664 16 3.65 0.30 17.28 211.65
128 32 3.56 0.22 17.74 290.65256 32 3.71 0.30 16.98 213.08512 64 3.60 0.48 17.53 130.31
1024 64 3.64 0.71 17.32 88.962048 128 3.75 1.28 16.84 49.31
MPI-I/O Read Performance for216,000 Ions of NaCl on XT5
University of EdinburghJanuary 2011
xyz,PDB
DL_FIELD
‘black box’FIELD CONFIG
DL_FILED
• AMBER & CHARM to DL_POLY
• OPLSAA & Drieding to DL_POLY
Protonated
University of EdinburghJanuary 2011
MgO, NaCl, simple ionic crystals - easyForce fields for
Two vdw descriptions: Mg2+ with O2- and O2- with O2-
Materials Force Filed
University of EdinburghJanuary 2011
Simple covalent molecules - bearable
For example: dimethyl sulphoxide
atoms 10HD 1.00797 0.09000 1HD 1.00797 0.09000 1HD 1.00797 0.09000 1CD 12.01115 -0.14800 1HD 1.00797 0.09000 1HD 1.00797 0.09000 1HD 1.00797 0.09000 1CD 12.01115 -0.14800 1SD 32.06400 0.31200 1OD 15.99940 -0.55600 1
bonds 21harm 4 1 644 1.11000harm 4 2 644 1.11000harm 4 3 644 1.11000harm 8 5 644 1.11000harm 8 6 644 1.11000harm 8 7 644 1.11000harm 4 9 480 1.80000harm 8 9 480 1.80000harm 9 10 1080 1.53000-126 1 8 96195 129.03-126 1 10 33433 84.716
angles 15harm 1 4 2 71.00000 108.40000harm 1 4 3 71.00000 108.40000harm 1 4 9 92.20000 111.30000harm 2 4 3 71.00000 108.40000harm 2 4 9 92.20000 111.30000harm 3 4 9 92.20000 111.30000harm 5 8 6 71.00000 108.40000harm 5 8 7 71.00000 108.40000harm 5 8 9 92.20000 111.30000harm 6 8 7 71.00000 108.40000harm 6 8 9 92.20000 111.30000harm 7 8 9 92.20000 111.30000harm 4 9 8 68.00000 95.00000harm 4 9 10 158.00000 106.75000harm 8 9 10 158.00000 106.75000
Also 12 dihedrals, and 10 vdw
Biological Force Field
University of EdinburghJanuary 2011
How about this?
Protein molecules consist of hundreds of amino acids.
4382 atoms19400 two-body7993 three-body13000 four body730 vdw
Clearly, manual entry is not practical!
Biological Force Field
University of EdinburghJanuary 2011
Two Temperature Model
• Superheating? Lattice temperature above the melting temperature with no track formation1. Continuum models cannot give accurate track radii; require MD-coupling!
1. D.M. Duffy, N. Itoh, A.M. Rutherford and A.M. Stoneham, J. Phys: Condens. Matt. 20 (2008), p. 082201
• Continuum neglects lattice straining and emission of shock waves (carry away some energy). CL in continuum is neither identical with Cv or Cp.
• Volume change by a phase transition is not included, i.e. Silicon shrinks upon melting
TTM+MD 50x50x10 unit Si cells 50keV/nm ~1ps
University of EdinburghJanuary 2011
• MD additions:• Coarse-grained T-cells• Choice of boundary conditions…
• Periodic for the atomistic lattice• Electronic system: z-dir von-Neumann, xy-dir infinite sink
• Inhomogeneous Langevin thermostat• Mechanism for electronic energy transfer to the lattice• Depends on the local electronic temperature
Molecular dynamicsContinuum
or
Towards Molecular Dynamics
University of EdinburghJanuary 2011
DL_POLY_4 Calling Tree
Always Advise with the MANUALAlways Advise with the MANUAL
Start up
Parse INPUT data
Set Limits & Array Bounds
Allocate Arrays
Read INPUT data, VERIFY & Make SENSE (semi-iterative)
Initialise/Read Restart data
Set Halo
Decomposition Bookkeeping, VERIFY & Report
Set Temperature/Motion (CS+CB+RB+CGM)
Main MD Loop (Time & Steps Conditioned)
Finalise & Report Statistics
Shut Down
University of EdinburghJanuary 2011
DL_POLY_4 Calling Tree
Main MD Loop (Time & Steps Conditioned) LFVMain MD Loop (Time & Steps Conditioned) LFV
Evaluate Inter-molecular Forces & Collect Spatial Statistics (VNL)
Make Adjustments & Evaluate Intra-molecular Forces
Leap Frog Verlet
RELOCATION & BOOKKEEPING & Set Halo
Collect Statistics
Print OUTPUT data
Check on Loop Condition
University of EdinburghJanuary 2011
DL_POLY_4 Calling Tree
Main MD Loop (Time & Steps Conditioned) VVMain MD Loop (Time & Steps Conditioned) VV
Velocity Verlet stage I
Evaluate Inter-molecular Forces & Collect Spatial Statistics (VNL)
Make Adjustments & Evaluate Intra-molecular Forces
RELOCATION & BOOKKEEPING & Set Halo
Velocity Verlet stage II
Collect Statistics
Print OUTPUT data
Check on Loop Condition
University of EdinburghJanuary 2011
DL_POLY on the Web
WWW: WWW: http://www.ccp5.ac.uk/DL_POLY/
FTP: ftp://ftp.dl.ac.uk/ccp5/DL_POLY/
DEV: http://ccpforge.cse.rl.ac.uk/gf/project/dl-poly/
FORUM: http://www.cse.scitech.ac.uk/disco/forums.shtml
University of EdinburghJanuary 2011
Acknowledgements
Thanks to
• Bill Smith (retired)• Chin Yong (STFC – DL)• Michael Seaton (STFC – DL)• Szymon Draszewicz (UCL)• Ian Bush (NAG Ltd.)