QRing − a scaleable, parallel software tool for quantum transport
simulations in carbon nanodevices using linear solvers on Intel
Xeon/Phi and GPU architectures
Mark A. Jack∗1, Benjamin Prather†2, Adam Byrd1, Leon W. Durivage3, and Mario R.Encinosa1
1Florida A&M University, Department of Physics, Tallahassee, FL 32307.2Arizona State University, Department of Physics, Tempe, AZ 85281.
3Winona State University, Department of Physics, Winona, MN 55987.
June 4, 2015
Abstract
The ability of a nanomaterial to conduct charge is essential for many nanodevice applica-tions. Numerous studies have shown that disorder including defects and phononic or plasmoniceffects can disrupt or block electric current in nanomaterials. An accurate theoretical account ofboth electron-phonon and electron-plasmon coupling in carbon nanotube- and nanoring-basedstructures is key in order to properly predict the performance of these new nanodevices. Amulti-scale continuum approach developed by collaborator M.J. Leamy (Georgia Institute ofTechnology) forms the basis for reduced-order phonon modeling of carbon nanostructures. Thedevice Green’s function Gd(E), derived within a non-equilibrium Green’s function formalism(NEGF) will function as key input to determine all transport observables. The transmissionfunction T (E) may be calculated from Gd(E) by inverting the Hamiltonian matrix at each en-ergy step E. Integrated observables such as for example source-drain current ISD may then becalculated by integrating over T (E). Using the parallel sparse matrix library PETSc providestools to optimally divide memory and computational tasks required in inverting the Hamil-tonian matrix across multiple compute cores or nodes. The results at different energy levelsare integrated via a second layer of parallelism to obtain integrated observables. Because theinversion at each energy step dominates the run time and memory use of the code, it is im-portant that the code scales well to realistic system sizes. Several different direct and iterativelinear solvers and libraries were compared for their scalability and efficiency, including the In-tel MKL shared-memory, direct, dense solver, the MUltifrontal Massively Parallel direct Solver(MUMPS), and iterative, sparse solvers bundled with PETSc. The National Science Founda-tion’s Extreme Science and Engineering Discovery Environment (XSEDE) resource Stampede atthe Texas Advanced Computing Center as well as regional resources in Florida provided by theSunshine State Education and Research Computing Alliance (SSERCA) are used for develop-ment and benchmarking, while the Department of Energy’s Oak Ridge Leadership ComputingFacilities (DoE OLCF) like Titan and NICS/Beacon are used for physics production runs.
∗[email protected]†[email protected]
1
Introduction
The reliable electrical and magnetic manipulation of information and energy in nanostructures at
the 1- to 100-nm scale is crucial for the development of novel, nanoelectronic devices [1]. Such ma-
nipulation must be highly precise, which requires understanding such phenomena as Shottky barri-
ers at metallic contacts, dissipation, material defects, and contact with the substrate. Carbon-based
nanostructures promise to be ideally suited for inexpensive, fast and low-power next-generation
electronics (nanoelectronics) [2, 3, 4]. Practical predictions of device behavior must account for
electrical resistance due to the presence of phonons and due to the electrons’ inherent excitation of
phonons [5, 6]. Graphene-based allotropes such as toroidal carbon nanotubes [7] hold the promise
of completely new nanodevice [3, 4, 5, 6, 8] and metamaterials applications [9, 10, 11, 12, 13, 14]. In
addition to persistent current and Aharonov-Bohm effects [15, 16], new electromagnetic moments
such as the toroidal moment will be generated by ring currents in a carbon nanotorus driven by a
voltage between two metallic leads [17, 18, 19, 20]. An example of the device geometry is sketched in
Fig. 1 including a tightbinding-derived model for the magnetic flux dependence of the source-drain
current (b). Another quickly expanding area of nanodevice applications is nanoplasmonics [21],
Fig. 1: a) Sketch of nanotorus resting on two metallic leads. b) Aharonov-Bohm oscillations ofcurrent vs. flux in a magnetoresistance plot.
where coherent electronic oscillations, plasmons, can be tuned via gate voltage and doping. In a
metamaterial of nanorings for example the interference of ring-generated electromagnetic radiation
from electronic surface currents driven by an incoming polarized wave front can lead to new opti-
cal response characteristics [9, 10, 11, 12, 13, 14]. Electronic correlation and exchange effects on
quantum transport, including exciton-phonon and exciton-plasmon coupling, are described effec-
tively in a tightbinding transport description using a non-equilibrium Green’s function formalism.
A multi-scale analysis tool was developed, which (1) rigorously quantifies these interactions, and
(2) yields open source tools available to the research community which allow general geometries
to be analyzed for inelastic effects in charge transport. Transport observables like transmissivities,
2
integrated currents, and magnetoresistance may be calculated via NEGF for realistic system sizes.
Transmissivities, integrated currents, and magnetoresistance might be sensitive to particular, sup-
pressed vibrational modes. The utilized model will permit the expected temperature-dependence
of resistance/conductivity and power loss to be expressed in the technically very important long
phonon-wavelength regime in nanoelectronics. Having in mind applications in the very new area
of nanoplasmonics on graphene-based architectures [21], electronic correlation and exchange effects
on quantum transport including electron-plasmon coupling can be described in a Hubbard model
[22, 23, 24] that allows a generalization of tightbinding transport calculations using NEGF. The
use of fast, parallel, sparse-matrix libraries will allow expansion to beyond tightbinding accuracy.
Transport observables may then be calculated quickly and efficiently for realistic system sizes of
10,000 atoms and more.
A curved manifold-based approach is employed in the electronic structure, phonon disper-
sion, and electron-phonon interaction modeling [25, 26]. In a collaboration with Georgia Tech’s
M.J. Leamy and Florida A&M University’s M. Encinosa the first two, decoupled models have al-
ready been completed using closely-related techniques for carbon nanotubes and nanotoroids. A
recently developed multi-scale continuum approach (M.J. Leamy) forms the basis for reduced-order
phonon modeling of carbon nanostructures. Electron-plasmon and exciton-plasmon couplings in
carrier transport may be incorporated as well in an effective quasi-particle tightbinding description
after appropriate transformation (Boguliobov canonical transformation) [24] and diagonalization of
the Hamiltonian containing a free exciton, a free plasmon and an exciton-plasmon interaction term.
The C++ code has an object-oriented and modular structure using MPI parallelism and sparse
matrix software libraries such as PETSc. PETSc includes routines to manipulate large sparse
matrices quickly and efficiently using an optimal distribution of the memory allocation for the
matrix over multiple cores. The algorithm is setup in such a way that the transmission function
T (E) may be calculated at a single energy E by inverting the Hamiltonian matrix to obtain the
transport Green’s function via the use of PETSc on multiple compute cores. Integrated observables
such as the source-drain current ISD for example can then be calculated by integrating over the
transmission function values for a pre-defined energy range using a second layer of parallelism
using MPI. High-performance computing resources are available via the new Florida-wide HPC
network SSERCA.org (Sunshine State Education and Research Computing Alliance), primarily
for code development and benchmarking, while large-scale simulation runs using 1000 and more
cores are conducted on national computational resources such as the National Science Foundation
XSEDE’s Stampede cluster at the Texas Advanced Computing Center (TACC) and the Department
of Energy’s Oak Ridge National Laboratory’s Leadership Computing Facilities (OLCF) Titan and
NICS/Beacon.
The research team of M. Jack and a number of summer students supported via NSF XSEDE
and BlueWaters/Shodor programs have developed a fast and versatile parallel software tool, QRing,
setout to estimate quantum scattering corrections in quantum charge transport in carbon nanos-
tructure devices effectively for realistic nanosystems modeling of up to 30,000 atoms using a tight-
binding, non-equilibrium Green’s function formalism. The code allows an effective and fast treat-
3
ment of first-order electron-phonon scattering corrections in single-electron transport for nanoscale
transistors based on straight or toroidal carbon nanotubes including static external magnetic fields
under a small bias potential for realistic device dimensions, which cannot be obtained with typical
ab-initio techniques such as density functional theory. A multi-scale continuum approach developed
by collaborator M.J. Leamy at Georgia Institute of Technology’s Department of Mechanical En-
gineering forms the basis for reduced-order phonon modeling of carbon nanostructures. Arbitrary
low-energy phonon modes important for nanodevice performance can be modeled and are read into
the code to incorporate electron-phonon scattering corrections.
The code has a modular organization, programmed in C++, using an object-oriented approach
to store the effective Hamiltonian for the device region as a large, sparse, complex and banded
matrix over the shared memory of individual nodes. The use of built-in MPI parallelism and a
state-of-the-art high-performance computing library, PETSc, solves for the electronic Green’s func-
tion of the system via fast, parallel inversion of the Hamiltonian matrix with which any transport
observable or band-structure characteristic can be computed. Using the parallel sparse matrix
library PETSc [27], PETSc provides tools to optimally divide memory and computational tasks
required in inverting the Hamiltonian matrix across multiple compute cores or nodes. Thus, mul-
tiple different nanodevice settings such as graphene nanoribbons, carbon nanotubes with ’kinks’
or non-closing nanorings can be just as easily implemented as lattice defects, impurities, and sim-
ilar scattering effects by a simple re-definition of the Hamiltonian matrix in the code. Equally,
new quasi-particle effects such as excitons and plasmons with exciton-plasmon coupling can be
investigated with an adjusted device Hamiltonian in a Hubbard model and effective tightbinding
description. Because the inversion at each energy step dominates the run time and memory use of
the code, it is important that the code scales well to realistic system sizes.
Our simulation code QRing can be considered a complimentary software tool in the space of
nanoscale device and quantum transport simulations. In this context, state-of-the-art programs
such as the tools OMEN and NEMO 3D created at Purdue University with petascale simulation
capabilities come to mind [28, 29, 30, 31, 32, 33]. In particular, OMEN provides modeling capability
for transport primarily in nanowires of III-V semiconductor materials in form of a multi-level
parallelism code for typical cross sections, gate lengths and body thicknesses of a few 10ths of
nanometers [28, 29, 30, 31]. Some recent work has also been conducted for graphene nanoribbons
[32, 33]. Purdue’s simulation tool can perform typical simulations in few minutes on up to roughly
200,000 compute cores. In comparison, our tool QRing focuses on carbon nanotube- and graphene-
based devices with comparable tube and ring widths of a few nm and tube lengths or torus diameters
of 40− 80 nm. QRing has been tested and shows excellent scaleability on up 16,000 compute cores
for system sizes of at least 24,000 atoms with simulation runs conducted on Titan at OLCF in
under one hour. QRing has been tested on large-scale computer clusters provided by XSEDE and
OLCF and shows excellent scaleability for system sizes of up to 30,000 atoms with. There is no
restriction in speed-up by simply increasing the number of available cores. The only restrictions to
occur are due to having to limit simulations to less than roughly 30,000 atoms due to maximally
available shared memory per compute core in order to avoid increased latencies from inter-node
4
communications.
In addition, OMEN models phononic corrections in a self-consistent tighbinding and NEGF
formalism with a Poisson solver but does not directly contain a module to treat other quasi-
particle corrections such as exciton-plasmon coupling, a core feature to be integrated with the
existing description for electron-phonon coupling in QRing as described further in this proposal.
Matrix elements for multi-electron scattering and correlation effects are currently being determined
by collaborators J. Jakowski at University of Tennessee Knoxville and J. Oelgoetz at Austin Peay
State University using the new technique density-functional tightbinding [34, 35, 36, 37] and will
be included in QRing in the next few months before the suggested proposal start date of July
1, 2015, thus providing a similar level of precision of theoretical prediction as tools like OMEN
but with additional quantum corrections. Due to its modular structure and reliance on state-
of-the-art libraries such as PETSc as described above these effects may be incorporated easily
in QRing by a redefinition of the Hamiltonian in tightbinding approximation. Also higher-order,
beyond tightbinding calculations can be done by appropriately defining additional off-diagonal
matrix elements with matrix calculations handed off to PETSc’s linear solver algorithms. QRing
simulation runs will be compared to existing thermoelectric experimental data on carbon nanotubes
provided by S. Khondaker’s group at the University of Central Florida’s NanoScience Technology
Center. Semi-empirical input parameters such as an electronic hopping term can be fine-tuned
to e.g. adjust for resistive effects at the contact sites between device region and metallic leads,
currently not treated theoretically at an atomistic level in the code, to more accurately estimate
the experimental results.
1 Theory and computational methodology
A research collaboration composed of faculty at Georgia Institute of Technology (M.J. Leamy)
and Florida A&M University (M. Jack) has developed two, decoupled models to describe phonons
on the one hand and electron transport on the other using closely related techniques for carbon
nanotubes and nanotoroids. A recently developed multi-scale continuum approach by Leamy [25,
26] forms the basis for reduced-order phonon modeling of carbon nanostructures (see Fig. 2). This
approach employs intrinsic basis vectors defined on a reduced-dimension surface (manifold) of the
nanostructured material. Changes in interatomic potential energy arising from lattice vibrations
are equated to changes in continuum strain energy, allowing non-quantum atomistic behavior to
be captured using continuum techniques. A subsequent finite element discretization results in a
significant decrease in the deformation degrees of freedom present, while at the same time accurately
capturing the acoustic range of the phonon spectra. Results for phonon modes show very good
agreement between the multi-scale approach and experimental results and ab-initio calculations
[38, 39].
5
Fig. 2: a) Continuum computation approach used to study reduced-dimension materials. b) Typicalphonon mode predicted using the continuum approach.
1.1 Non-equilibrium Green’s function method (NEGF)
All observables can be derived from the (advanced / retarded) Green’s function G(a,r)d (k) of the
device region [40, 41, 42]: [E(k)I −H(k)− ΣL − ΣR ± iη
]G
(a,r)d (k) = I. (1)
The effects of the left (right) metallic lead are folded into an effective Hamiltonian shown in Eq. 1
in form of self-energy corrections ΣL,R with H(k) describing the device region. I symbolizes the
unit matrix. In tightbinding approximation, only the overlap integrals of the nearest-neighbor pz-
orbitals are considered and are described in terms of a general electronic ’hopping’ parameter te
for transmission along the hexagonal lattice structure. With the determined Green’s function, the
current ISD can be calculated with the transmission function (transmissivity) T (E) via:
T (E) = Trace[ΓLG
rΓRGa], (2)
ISD =2e
~
∫dE T (E)
[fL(E)− fR(E)
]. (3)
with fL,R(E) as Fermi distributions at the two contacts and ΓL,R as nanotube-to-lead couplings
(leads are held at thermodynamic equilibrium while transport in the device region is out-of-
equilibrium). One can calculate density-of-states D(E) and transmission function T (E) for electron
transport through the torus between two attached metallic leads under a small voltage bias.
The effective Hamiltonian yields a sparse matrix with the non-zero elements mostly arranged in
a banded structure along the diagonal and at the corners of the matrix for torus closure [43, 44, 45].
To be exact, for example for a (10, 10) armchair or a (10, 0) zigzag chirality, due to the nanoring
(or nanotube) geometry with ring-to-ring coupling, a self-similar organization of the Hamiltonian
matrix occurs into a subset of, for example, 40×40 or 20×20 banded, sparse matrices, respectively.
6
1.2 Phonon-modulated electronic hopping term, electron-phonon coupling andphonon-modulated electron-electron correlation
The interaction of electrons with phonons in charge transport will be modeled using additional
terms in the Hamiltonian, which account first for a phonon-modulated electronic hopping term
te. We follow the theoretical description provided in [38, 39]. Low-energy phonon modes for
carbon nanorings of arbitrary size and chirality are calculated in a continuum model description
via a finite-difference-time-domain code (FDTD), provided by collaborator M.J. Leamy at Georgia
Institute of Technology’s Mechanical Engineering Department. Amplitudes of oscillations of atomic
coordinates for different low-energy phonon modes including breathing, stretching and torsional
modes are now included in our tightbinding transport calculations of transport observables such
as transmissivities, source-drain currents etc. The modified strength of the term te will be derived
from the lattice deformation accompanying each phonon mode and from the occupancy of the
phonon modes at the temperature of the system [25, 26]: First, an effective deformation potential
of the form V1 = g1
(uxx + uyy
);uxx = ∂ux
∂x + uzR ;uyy =
∂uy∂y will change the effective electronic
hopping term contained in the diagonal and off-diagonal matrix elements of the Hamiltonian.
ux,y,z are the local deformations of the torus in the continuum model; R is the the torus’ large
curvature radius. Parameter g1 is given as approximately 30 eV. The corresponding Hamiltonian
matrix elements are given in the literature [39]. The effect of phononic corrections may now be
captured in the Hamiltonian by superimposing a hexagonal lattice structure of carbon atoms onto
the deformed torus surface and replacing the electronic hopping parameter te,h = −3.1 eV by an
effective parameter te,hij due to the relative displacement of adjacent carbon atoms i, j during lattice
vibrations characterized quantum mechanically by phonon modes. In good first approximation,
te(h) is now enhanced by a term proportional to the atomic displacements δij = r′ij − rij and scales
linearly with the factor ε = 5.4 eV A to be derived in an atomistic calculation e.g. via a Morse
potential approach for the overlap of pz-orbitals of adjacent C atoms [38, 39]:
te(h)ij = te(h) + ε ηij δij with:
ηij =
(r′ij−rij
)∥∥∥r′ij−rij∥∥∥ ; δij = δi − δj ;
δij = r′ij − rij .
(4)
For a more advanced treatment of vibrational modes, following the description in [38, 39], the
Hamiltonian will also contain a free-phonon term and, more importantly, a term describing electron-
electron interactions, which will now be modulated by the displacements of the atomic cores. In
the rigid-ion approximation, valid for long-wavelength and thus low-energy phonons, the atomic
cores move adiabatically slowly to each atom’s electronic charge distribution so that the electron
distribution is modeled to instantaneously adjust to the position of the positive ion cores. Thus,
the oscillatory displacement of the atomic cores will lead to an effective oscillatory separation of the
charge clouds of neighboring atomic cores and thus to a change of the effective Coulomb repulsion
between different electrons. This effect can be treated by additional off-diagonal contributions V2
7
to the Hamiltonian matrix,
V2 = g2 ei3η(uxx − uyy + i2uxy
); 2uxy =
∂ux∂y
+∂uy∂x
; g2 ≈ 1.5 eV, (5)
and can be parameterized in form of two terms − a linear electron-phonon coupling term and
a phonon-modulated electron-electron correlation term. The matrix element for linear electron-
phonon coupling e.g. in a (10, 10) armchair carbon nanoring is listed here to include low-energy
(acoustic) phonons (small Q):
Mqz ,Qz = −4iMqz
(qηQ
)XQz ; Mq = νqρ
2e(q). (6)
The parameter g2 measures the relative strength of this term with respect to the deformation
potential contribution proportional to g1. νq and ρe(q) are the electron velocity and electron
density, respectively, at electron momentum q. Phonon frequencies ωph and phonon polarization
vectors η can all be directly extracted for each atomic lattice site from the finite-element based
simulation by properly discretizing the continuum-model description of the ring distortions in the
finite-element based simulation. For a given vibrational mode, this information will then be read
into the electronic transport Hamiltonian for each matrix element.
1.3 Inclusion of multi-electron correlations via density functional tightbinding
In collaboration with the University of Tennessee Knoxville and Austin Peay State University, re-
searchers J. Jakowski and J. Oelgoetz will provide the necessary matrix elements for electron-phonon
coupling and phonon-modulated electron-electron interaction via a new methodology called den-
sity functional tightbinding (DFTB) [34]. In DFTB, both electronic and a subset of (light) nuclear
coordinates may be treated quantum mechanically with an exact, DFT-like treatment of valence
electrons while wave functions of the lighter nuclear coordinates are estimated approximately with a
reduced basis set expansion with the heavier nuclei treated classically [35, 36, 37]. A self-consistent
treatment of electron density in DFTB can be used routinely in dynamics of roughly 1000 atoms.
For calculational simplicity and efficiency, a non-self-consistent treatment will be used for larger
systems from which the device Green’s function is evaluated at each energy. A self-consistent de-
scription may be included at a later stage. While a full MD-type calculation for torus sizes of
several 30,000 atoms and more is not possible in DFTB, the plan is to calculate the corrections for
a short segment of e.g. a (10, 10) torus chirality consisting of a few connected small-diameter rings
of a 1000+ atoms and then to scale these to the full torus dimensions using symmetry arguments
of point groups of a graphene lattice compactified to a toroidal surface with chiral indices (m,n).
Jakowski possesses a dual appointment as researcher with ORNL and as XSEDE staff member
with computational resources readily available to both him and Oelgoetz at ORNL (Titan), NICS
(NICS/Beacon) and XSEDE (Stampede, Darter). This collaborative activity is conducted as an
XSEDE Extended Collaborative Support Service (ECSS) with Jakowski acting as XSEDE staff
support. A list of tasks to be completed is sketched below:
• QRing code ported to NSF XSEDE resources;
8
• Interface between standalone QRing and DFTB programs;
• Validation of QRing/DFTB approach for quantum transport;
• Analysis of performance and optimization of QRing/DFTB;
• Integration of DFTB and QRing into a single binary.
An expansion of this research program to include other scattering corrections such as lattice
defects, adsorbed atoms, etc. via DFTB is planned. The overall model would allow to generally
compute the expected temperature-dependence of resistance/conductivity and power loss in the
technically very important long-phonon-wavelength regime, which prototypical carbon nanostruc-
tures will experience in next-generation electronics. Results from the model will be compared
with experimental data made available by S. Khondaker at the University of Central Florida’s
NanoScience Technology Center.
1.4 Experimentally realistic metallic lead geometry
Tight-binding transport calculations typically rely on a one-to-one identification between lattice
sites comprising the contact layer of the metallic lead to the carbon atoms of the device region.
A major obstacle that must be overcome when a toroidal nanotube is placed atop a rectangular
semi-infinite lead (see Fig. 1) is that the alignment between metallic and carbon sites is no longer
maintained. The locations of the carbon atoms comprising the tube/torus trace an arc with some
carbon atoms in direct contact with the lead while the lattice sites of the metallic lead are generally
taken to comprise a rectangular lattice. The straightforward calculation of the Green’s function
G(r, r′
)for transport along one semi-infinite metallic lead yields 1
G(r, r′
)= − 16me
π~2LyLz
∑m,n
sin(mπyLy
)sin(mπy′Ly
)sin(nπzLz
)sin(nπz′Lz
)
·+∞∫−∞
dksin kx sin kx′(
k + kmn + iη)(k − kmn − iη
) (7)
with k, kmn trivially related to the energy E and particle in a box energy eigenvalues Emn through
kmn =
√2me
(EFermi − Emn
)/~ (8)
In one approach, in order to obtain the Green’s function values, the longitudinal degree of
freedom along the leads is treated as discrete, and after binning, the coordinates of the carbon
atoms in contact with a metallic lead are searched which are closest to metallic lattice sites. The
metallic Green’s function value may then be computed at that site as an approximation for use
in the device Hamiltonian. As this approach can be criticized as being time-inefficient and would
artificially introduce a resolution of accuracy into the calculation that is unwarranted and arguably
1This configuration corresponds to the original, computationally simpler implementation with the metallic leadstouching the torus region laterally on both sides in a few atomic contact sites.
9
outside of the scope of the method, an alternate approach was adopted: It is well known that the
expression listed above produces a Green’s function that is infinite when x = x′ which map into
diagonal elements of the matrix for the device region, Gd. The goal was to produce a hybrid method,
one that keeps the longitudinal degrees of freedom continuous to allow for a trivial matching to
carbon sites, but remains finite when x = x′. This is accomplished by introducing integration limits
that correspond to the cutoff momentum kcut of a lattice of spacing a , i.e. kcut = πa . The integral
can then be evaluated in very good approximation by contour methods to yield the result
Gcut
(r, r′
)= − 8me
~2LyLz
∑m,n
sin(mπyLy
)sin(mπy′Ly
)sin(nπzLz
)sin(nπz′Lz
)
·exp
(ikmn x>
)sin(kmn x<
)kmn
(9)
Only poles that fall within the cutoff kcut = πa contribute to this Green’s function. The argument of
the exponential becomes imaginary when Emn > EFermi, which allows for a controllable convergence
criterion. For typical values of EFermi and Emn, suitable convergence is obtained with on the order
of 1000 summation terms for each index m,n if the lead has a square profile, or less, should the
lead be flatter along the z-direction.
1.5 Exciton-plasmon coupling as example of advanced quasi-particle corrections
The implementation of a new, upgraded Hamiltonian into the existing code to include plasmonic ef-
fects e.g. in form of electron-plasmon coupling terms and eventually expanded to an exciton-plasmon
is also envisioned in the near future. This will entail a significant reorganization and expansion of
the part of the code that defines the distributed Hamiltonian matrix, which is anticipated to take a
few months with the necessary testing and debugging phase of the code. A schematic of how a new
tightbinding Hamiltonian H for strong exciton-surface-plasmon coupling (as demonstrated in semi-
conducting and metallic carbon nanotubes) can be constructed via a quasi-particle description is
illustrated below with H = Hplasmon+Hexc+Hint and Hplasmon, Hexc, and Hint as the free-plasmon,
free-exciton and interaction terms, respectively [22, 23, 46, 24]:
Exciton-plasmon interaction:
Hint = − e
mec
∑n
A(n)[
pn −e
2cA(n)]
+∑n
d(n)∇nφn. (10)
Boguliobov canonical transformation (diagonalization for quasi-particle description):
Hint =∑
k,µ=1,2
~ωµ(k)· (k)ξµ
(k)
+ E0. (11)
2 History and early benchmarks on TACC/Ranger cluster
Historically, a much simplified version of the code existed as a serial Fortran program developed by
co-investigator M. Encinosa at the NASA Ames Research Center in the summers 2000 and 2001
10
to study charge transport in armchair carbon nanotubes. A fast recursive algorithm to calculate
the transport Green’s function for system sizes of up to 4,000 atoms had been implemented and
was extended to include armchair nanoring devices for initial physics studies of transport under
magnetic flux. Typical calculations of ISD curves and magnetoresistance plots (ISD vs. B) for
(3, 3) armchair carbon nanorings with 3600 atoms (20 nm diameter) completed in about 25 hours
on a single processor of a computer work station. With the need to be able to model realistic
device geometries with nanorings of up to 200 to 500 nm in diameter, it was decided that a faster
and more modular parallel algorithm would be necessary. A project emerged to completely re-
write the simulation in form of a newly reorganized C++ program with an object-oriented and
modular structure. An object-oriented C++ code with MPI parallelism and the use of sparse
matrix software libraries such as PETSc [27] was developed by M. Jack and summer students L.W.
Durivage (Winona State U.) and A. Byrd (Florida A&M U.) in summers 2010 and 2011 as part
of the BlueWaters/Shodor Undergraduate Petascale Education Program (UPEP). PETSc contains
a range of routines in C++ using a strongly modular and object-oriented approach to handle
large linear systems. The PETSc library itself, and through the use of add-on packages such as
MUMPS [47, 48] includes routines to calculate large sparse matrices quickly and efficiently using
an optimal distribution of the memory allocation for the matrix over multiple cores. This modular
approach was also used in the organization of the transport code itself together with PETSc’s
parallel inversion routines to be able to later include more advanced transport modeling beyond
tightbinding and for excitonic and/or plasmonic scattering corrections via simple replacement of
the effective Hamiltonian [22, 23, 46, 24].
The algorithm is setup in such a way that the transmission function T (E) may be calculated at
a single energy E by inverting the Hamiltonian matrix to obtain the transport Green’s function via
the use of PETSc on multiple compute cores. The source-drain current ISD for example can then
be calculated by integrating over the transmission function values for a pre-defined energy range
(depending on the source-drain voltage), see section II. For an accurate estimate of integrated ob-
servables such as ISD with up to 3 decimal values numerical accuracy, T (E) values are calculated
for a grid of roughly 8,000 energy values E between −0.2 and 0.2 eV. These 8,000 values are cal-
culated via a second layer of parallelism using MPI in an embarrassingly parallel fashion. Through
increase of the number of cores for each distributed PETSc/MUMPS [47, 48] matrix inversion also
the overhead in communication between cores will increase, slowing down the overall calculation.
Thus, the number of cores necessary to do a single matrix inversion should be kept to a minimum
depending on the memory requirements to store that matrix. If the total processor number is not
an integer multiple of the number of cores in each group handling a single PETSc/MUMPS matrix
inversion at a single energy then all the processors are divided into groups of equal number of
cores while the remaining cores are arranged into one smaller group for one additional energy. The
computation circulates between the different processor groups in such a way that when a group
has completed a T (E) calculation that group receives the next available energy value but waits
until all the groups have completed their calculations. The T (E) values are stored using MPI calls
on one core of each group (sub rank 0) and then stored with one core (global rank 0) in a single
11
array. After all 8,000 T (E) values have been calculated and stored, the processor of rank 0 then
performs the final numerical integration over the stored (E, T (E)) pairs to calculate the total cur-
rent. Benchmarks were all conducted on different high-performance computing resources in Florida
available via the new HPC network Sunshine State Education and Research Computing Alliance
(sserca.org). Details on facilities and available compilers, libraries and software may be found in
the supplementary documentation added.
The plots in Fig. 3 illustrate clearly the nearly optimal scaling behavior of the code for different
levels of MPI parallelism and parallel and distributed Hamiltonian matrix evaluations via PETSc
and MUMPS [47, 48]. The scaling behavior of the code, i.e. the exponential decrease in compute
time with the increase in number of employed compute cores, is demonstrated where the PETSc
and MUMPS libraries are used by each individual core or in a distributed way over two-core groups
per energy step, i.e. the Hamiltonian matrix is inverted for each energy step on a single core or
distributed onto two cores. The scaling behavior is shown for runs on up to 64 or 128 cores,
respectively. The close to perfect scaling behavior for the simulations with two-core versus single-
core matrix inversions can nicely be seen in the logarithmic plots in Fig. 3 for both the use of
the MUMPS and PETSc libraries with visible significant improvement in timing switching from
the MUMPS to PETSc solvers. The system of choice for the benchmarking results are clusters at
Florida State University’s RCC Center as part of Florida’s Sunshine State Education and Research
Computing Alliance (SSERCA) but similar timing for armchair and zigzag nanoring simulations
was observed on the Pegasus cluster at University of Miami’s RCC Center.
Plots
Page 2
1 2 4 8 16 32 64 128
00
8640
17280
25920
34560
43200
Cntor PETSC vs MUMPS Comparison
Armchair
MUMPS Solver – Average MUMPS Solver – Perfect Scaling
PETSC Solver – Average PETSC Solver – Perfect Scaling
Number of Processors
Tim
e (
Se
co
nd
s)
1 2 4 8 16 32 64 128
4
5
6
7
8
9
10
11
Cntor PETSC vs MUMPS Comparison
Armchair
MUMPS Solver – Average MUMPS Solver – Perfect Scaling
PETSC Solver – Average PETSC Solver – Perfect Scaling
Number of Processors
Ln
Tim
e (
Se
co
nd
s)
Plots
Page 8
1 2 4 8 16 32 64 128
00
4320
8640
12960
17280
21600
25920
Cntor PETSC vs MUMPS Comparison
Zigzag
MUMPS Solver – Average MUMPS Solver – Perfect Scaling
PETSC Solver – Average PETSC Solver – Perfect Scaling
Number of Processors
Tim
e (
Se
co
nd
s)
1 2 4 8 16 32 64 128
3
4
5
6
7
8
9
10
11
Cntor PETSC vs MUMPS Comparison
Zigzag
MUMPS Solver – Average MUMPS Solver – Perfect Scaling
PETSC Solver – Average PETSC Solver – Perfect Scaling
Number of Processors
Ln
Tim
e (
Se
co
nd
s)
Fig. 3: Close to perfect scaling of tightbinding code using the PETSc or MUMPS linear solvers onone or two cores for a single matrix inversion at one energy step E. a) Left column of plots: FSUHPC. b) Right column of plots: Univ. of Miami HPC (Pegasus).
The starting tight-binding Hamiltonian is a sparse matrix containing mostly zeros, with the
12
nonzero elements mostly arranged in a banded structure along the diagonal and at the corners of
the matrix. The diagonal and corner matrix elements can be arranged in smaller matrices that
show the same banded structure as the large matrix describing the full Hamiltonian. The additional
corner elements in each case provide ring closure for both the small and large circumference of the
nanoring. (For the simpler carbon nanotube arrangement, those corner elements in the larger matrix
can be set to zero. For a graphene nanoribbon device, in principle the corresponding simplifications
occur for the smaller matrices occur due to removal of the periodic lattice condition also for the
small ring circumference but a modified Hamiltonian describing an effective linearized Dirac-like
electronic behavior on a graphene lattice is more appropriate there.) It is important to note that in
spite of the Hamiltonian matrix H being sparse and banded, the computational problem will scale
with a factor N2 with N as the number of atoms. The PETSc/MUMPS routines utilize two dense
(N,N) matrices to invert H.
3 Library studies on Intel Xeon/Phi and GPUs
3.1 Computational overview
One of the key tasks during this summer 2014 was to work on the code’s scalability and opti-
mize memory usage on different computer architectures using NSF XSEDE, OLCF and regional
resources. The goal was to keep the wall clock time per simulation run of about 8,000 large, sparse
matrix inversions at a manageable level of a few hours per simulation run so a range of different
nanoring chiralities and different low-energy phonon modes maybe studied in a single simulation
run at realistic device dimensions. Physics student Ben Prather at Arizona State University par-
ticipated in the project remotely as part of the 2014 NSF XSEDE Summer Student Engagement
Program under M. Jack as summer research mentor. The effective implementation of optimized
parallel linear solvers such as the PETSc library were studied to optimize code performance in
terms of optimal load balancing, memory usage, speed-up and scalability for system sizes from
roughly 10,000 up to 30,000 atoms. The linking with libraries such as Intel MKL and new tools
like MAGMA [49, 50], developed at the University of Tennessee Knoxville, was investigated with
off-loading of matrix calculations onto Intel Xeon/Phi co-processor cores. Key technical constraint
is that more than 90% of the run time and memory usage of the program consist of these in-
versions. To get an idea of the memory requirements for such a problem size, storing a single
Hamiltonian (or its inverse) as a complex, dense matrix as required by PETSc’s setup e.g. requires
24, 000 × 24, 000 × 16 Bytes = 9.2 GB of memory for a Hamiltonian of 24,000 atoms. An equally
memory-intensive step follows for the actual matrix inversion by direct solution after LU factor-
ization, performed at each energy level. The individual matrix inversions at each energy step can
be treated as independent problems and are parallelized using MPI. Inter-node communication is
to be avoided due to the much longer latencies involved, and matrix calculations are thus to be
performed using all available memory per node.
Currently, there does not exist a library that provides good support for the Intel Xeon/Phi
(MIC) coprocessors in performing all LAPACK functions. Thus, we created a comparative study
13
using five different configurations for linear solvers: a) PETSc with iterative GMRES solver; b)
PETSc with direct MUMPS solver; c) dense solver routines called from the Intel MKL library
linked to PETSc, run entirely on the processor cores; d) off-loading some matrix calculations in
Intel MKL onto the Xeon/Phi co-processor cores; and e) to perform all work on the co-processor
cores via the use of the MAGMA MIC library [49, 50]. For the use of parallelized libraries (PETSc,
Intel MKL, MAGMA MIC), there are two basic strategies for storing a matrix:
• Dense, in which each element receives its own memory, regardless of what value it contains;
• Sparse, in which only a list of nonzero elements and their indices are stored.
Additionally, there are two types of linear algorithms that can be distinguished that either use
a direct solution method with a dense representation of the original matrix and a subsequent LU
factorization step to invert it in place or an iterative solver that uses an initial guess of the solution
via an initial pre-conditioning step for the matrix and then iteratively refines this guess to the
actual solution within some defined error bounds. The timing analysis using MAGMA was limited
to small system sizes of less than 10,000 atoms as the memory available for complete offloading
onto the MIC is limited to 8 GB. Like most matrices, once the Hamiltonian is inverted it contains
very few zeros. Hence, the lower limit on memory use per matrix inversion is that taken by the
stored ’inverse’ Hamiltonian (Green’s function) of 9.2 GB.
There are three classes of algorithms for solving a matrix, once it has been stored:
• Pure dense solvers use a dense representation of the original matrix, then use an LU fac-
torization step and then invert it in place. Since they assume the original matrix could be
anything, they read and multiply all of its zero elements unnecessarily
• Direct solvers try to solve the system A × B = I for B, which is the inverse. In addition to
the initial matrix A, which can be stored sparsely, a direct solver requires storing the solution
B and the identity I as dense matrices , doubling our base memory requirement to 18 GB.
However, it can be faster than a pure dense solver, since it reads only the nonzero values from
A and uses them for the computation.
• Iterative solvers are set up just like direct ones, but use an initial guess for matrix B and then
refine the initial guess until the resulting product A × B is (within some acceptable error)
equal to I. Because the time that is spent for these methods to converge varies based on the
individual algorithm, the pre-conditioner used and the actual sparseness of the initial matrix,
they can be faster or slower than any of the other methods.
Four different configurations were studied:
• PETSc-MUMPS: direct solver, the current method used for the inversions;
• PETSc-GMRES: iterative solver using the same framework;
• MKL: pure dense solver using the LAPACK implementation in the Intel MKL;
14
• MAGMA: pure dense solver using the LAPACK implementation in MAGMA-MIC.
The results for the four studies were the following:
• PETSc-MUMPS: Because this solution requires 18.4 GB of memory, only one inversion prob-
lem fits on each node of Stampede.
• PETSc-GMRES: Memory requirements are the same as for MUMPS, but the solution times
for the GMRES iterative solver are dramatically lower.
• MKL: Memory is halved when using both pure direct methods, to a mere 9.2 GB. Thus
three problems fit on one node instead of two. Scaling works well, but currently off-loading
calculations onto the MIC is available for the LU factorization.
• MAGMA: System sizes with 24,000+ atoms do not fit into the memory for a MIC.
Additionally, a number of other updates and improvements to the code’s user-friendliness were
incorporated: The code was ported to version 3.4 of PETSc, which changed some function calls’
formats and required an explicit call for pre-allocating memory of sparse matrices. Command-line
flags were added to the user may easily set the number of atoms, the number of processes or the
number of threads per single matrix inversion and other parameters. At the end of each simulation
run, the code integrates the electronic transmission rate T (E) (obtained from the Green’s function
after each H inversion) over a range of energy levels E to obtain integrated observables such as
the source-drain current. By reading-in those T (E) values from a file rather than storing them in
memory and by allowing the code to be started at any energy level via a command-line flag, simple
checkpointing is enabled to store intermediate results if the code should crash or stop. The user
may specify the starting energy for a resumed simulation, and the new values will be appended to
the same output file that stores all T (E) values for subsequent integration.
The initial code development was conducted on a SSERCA cluster (Florida State University
RCC). New code builds were compiled for small system sizes to test sparse and dense parallel linear
solvers of the HPC libraries outlined earlier. The Stampede cluster at the Texas Advanced Com-
puting Center then acted as primary resource to perform these scalability and performance tests
for all system sizes using different linear dense and sparse solvers, direct and iterative methods,
multi-threading and off-loading onto Intel Xeon/Phi nodes. Stampede has nodes with 32 GB avail-
able memory shared by 16 processing cores and one Xeon Phi coprocessor per node. An XSEDE
startup allocation on Stampede of roughly 70,000 SUs (SUs=service units) was available to conduct
these tests with a request for an extension by 50,000 SUs recently approved. Stampede possesses
standard 32 GB Intel Xeon/Phi nodes with GPU capability. The NICS/Beacon cluster managed
by the National Institute of Computational Sciences (NICS) at ORNL possesses 48 compute nodes
with 256 GB shared memory, 16 Intel Xeon E5 processors and 4 Intel Xeon/Phi co-processors with
8 GB memory each and is a perfect testbed for large-memory tests and a perfect bridge between
Stampede and Titan with their respective Xeon/Phi and GPU nodes. An additional allocation of
5000 node hours on NICS/Beacon was also available.
15
Fig. 4: Solution times for a single matrix inversion for different nanotorus dimensions and use ofdifferent solvers (here: (10, 10) armchair).
3.2 Benchmarks
The graph for time-stepping of individual matrix inversions for comparisons a) through d) is shown
in Fig. 4. The MUMPS solver for example is very sensitive to exact system sizes and the parallel
storage of the matrix on the cores of each node. For sizes of 12,000 or 24,000 atoms, MUMPS
performed competitively to Intel MKL’s dense solver, which only used half of the memory in the
MUMPS study. MUMPS requires, due to its internal setup, the additional storage of the unit
matrix as dense matrix to later store the dense solution. In the end, PETSc’s iterative solver
GMRES using only one processor core per node to utilize all the node’s shared memory was found
to be faster than any of the other methods listed by roughly a factor of 10, thus prompting our
decision to use this approach for all production runs on Titan.
During a Department of Energy sponsored faculty visit at Oak Ridge National Laboratory
(ORNL) this summer 2014, first production runs to simulate quantum transport including phonon
scattering corrections were started using the Titan resource at ORNL’s Leadership Computing
Facility (OLCF) courtesy of summer host B. Sumpter after an initial test phase in early June on the
OIC/Phase 5 cluster at ORNL’s Center for Nanophase Materials Sciences (CNMS). In the initial
test phase on OIC/Phase 5, simulation runs were performed for small system sizes (10,000 atoms)
using about 2 to 8 nodes with 16-core nodes before switching to OLCF/Titan for first successful
production runs for nanorings from 3,600 to 24,000 atoms and a range of different armchair and
zigzag chiralities. Titan’s general scheduling policy promoting computational jobs with very large
node count but compute times restricted to one-hour runs proves optimal for our production runs.
A sketch of typical node counts n per job on Titan for different system sizes N and estimated SUs
16
spent is listed below:
N = 3600− 12000; armchair: n = 64; zigzag: n = 128.N = 12000− 24000; armchair: n = 128− 256; zigzag: n = 256− 512.
Example: (6, 0) zigzag: N = 24000; n = 1024.Max. run time: 1 hour; SUs per run: 2048− 8192 SUs.
3.3 Physics production runs
Initial simulation runs for a range of armchair and zigzag chiralities and ring sizes nicely reproduce
van-Hove singularities in form of spikes in the density-of-states D(E) and transmissivity T (E). Van-
Hove singularities correspond to resonances in transmission of the interfering electronic pathways
on the torus surface with carrier injection and extraction from the attached metallic leads. To be
exact, 1D conductors will show spikes in the density-of-states D(E) that diverge like 1√E
at band
edges due to electronic confinement along the small torus circumference [3]. In Fig. 5a for example
the density-of-states are depicted with significant enhancements visible at resonance energies acting
as available transmission channels. Armchair chiralities in Fig. 5a are (3,3), (5,5) and (10, 10) for
N = 3, 600 carbon atoms. Rings along the minor circumference include 12, 20, and 40 atoms
and rings along the major circumference 300, 180, and 90 atom, respectively. Resonance energies
depicted in Fig. 5a clearly scale with 1/R with R being the major ring radius. These signatures
can clearly be reproduced with the new parallel code for all ring sizes and all armchair tori and
those zigzag tori that are metallic, i.e. conducting. The general prescription that carbon nanotubes
with chiralities (m,n), that satisfy mod[(m − n)/3] = 0, are metallic carries over to the toroidal
armchair (n, n) and zigzag (n, 0) cases. Simulations of zigzag rings with mod[(m − n)/3] = 1 or
2 do not show any transport but semi-conducting behavior as expected. The density-of-states
D(E) vanishes at the Fermi energy [3]. Additionally, width-dependent effects such as a splitting of
van-Hove singularities can be observed where transport resonance for metallic tori along the inner
or outer radius of the torus surface creates a splitting of a van-Hove singularity into two distinct
peaks leading to a drop-off in transmissivity T (E) by several orders of magnitude between peaks.
The effect is more pronounced with increasing torus width relative to the large torus diameter.
The splitting of resonances is highlighted in Fig. 5b for example for a (10, 10) armchair torus of
3,600 atoms: A van-Hove singularity at E = 0.187 eV transforms into two symmetric peaks with
a roughly 3 meV splitting, a 2%-effect for a ring width of 2a = 13.6 A and a torus diameter of
2R = 35 A.
4 Results
A summary of all benchmarking results for different linear solvers and libraries or different com-
puting models (only on processor cores, off-loading on Intel Xeon/Phi co-processors or GPUs, or
symmetric) completed with XSEDE summer student Ben Prather this summer and fall 2014 is
shown in Fig. 6 below. Simulations have been conducted both on NSF XSEDE (TACC/Stampede)
and DoE OLCF (Titan, NICS/Beacon, OIC/Phase5) resources for different nanoring dimensions,
17
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●●
●
●
●
●
●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
−0.2 −0.1 0.0 0.1 0.2
1e−
031e
−01
1e+
011e
+03
Energy E (eV)
Den
sity
D(E
)Density of states by energy at 3600 atoms
● ● ●
Chiralities
33 55 1010
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
−0.200 −0.195 −0.190 −0.185 −0.180
1e−
021e
+00
1e+
021e
+04
Energy E (eV)
Den
sity
D(E
)
Density of states by energy at (10,10), 3600 atoms
Fig. 5: a) Density-of-states D(E) with van-Hove singularities for (3,3), (5,5) and (10, 10) arm-chair nanorings (N=3,600). b) Splitting of a van-Hove singularity in (10, 10) armchair nanorings(N=3,600).
5000 10000 15000 20000 250000
50
100
150
200
250
300
350
400
450
Solution Time vs Problem Size for Various Solvers
GMRES
MUMPS-thd
MUMPS
MKL
MKL-MIC
MAGMA
MAG-MIC
Size of system (atoms)
Tim
e to
sol
utio
n of
one
ene
rgy
step
(s)
Fig. 6: Comparison of compute times per matrix inversion (single energy step) for different linearsolvers, libraries and computing models.
chiralities, device setups and phonon modes. All library benchmarks have been performed and
recorded this summer on Intel Xeon/Phi and GPU architectures for symmetric, heterogeneous and
offloading programming models to study optimal code performance for system sizes of 10,000 to
30,000 atoms. For our tightbinding sparse-matrix description the traditional GMRES iterative lin-
ear solver using one core and all the memory per node still proved to be the most effective approach
by far.
5 Conclusions
The new parallel code QRing correctly predicts quantum transport features in carbon nanoring
devices including van Hove singularities and width effects. It is scalable to armchair/zigzag systems
withN = 30000 atoms and versatile to model nanoring, nanotube, and graphene nanoribbon setups.
18
The general use of HPC linear solver libraries like PETSc allows for a later extension to higher-order
approximations beyond tightbinding. Following physics program is scheduled:
• To model low-energy phonon corrections with effective linear electron-phonon coupling pro-
vided in a collaboration with Georgia Institute of Technology (M.J. Leamy) via a continuum
method using a finite-difference time domain approach [25, 26];
• To place the nanodevice on top of metallic leads for o(100×) larger current amplitudes;
• To study the stability of modal symmetries in coherent transport under electron-phonon
coupling to low-energy phonon modes [51];
• To include magnetic-field dependence for magneto-transport studies;
• To extend to other quasi-particle corrections such as exciton-plasmon coupling etc. [22, 23,
46, 24].
On the technical side, it is planned to use available open-source and commercial parallel profiling
tools such as TAU to study parallel thread calls on Intel Xeon/Phi cards and how manipulating
these impacts the code performance, to test other iterative solvers with added pre-conditioning,
and to investigate further code improvements with PETSc version 3.5.0 linked to the new sparse
solver PARDISO [52, 53]. These studies may be continued with MAGMA on GPUs using CUDA
compilers and tools such as CUBlas.
Acknowledgments
Funding for this project was provided in parts by the 2014 XSEDE Summer Student Program
(BP) and the Shodor/Blue Waters Undergraduate Petascale Education Program (AB, LD). MJ
would like to acknowledge the support by the 2014 ORISE Visiting Faculty Program at Oak Ridge
National Laboratory (ORNL) funded by the Department of Energy. MJ would further like to
thank the ORNL Center for Nanophase Materials Sciences and the Sunshine State Education
and Research Computing Alliance (SSERCA) for available computational resources and helpful
discussions. The team would like to thank in particular collaborators M.J. Leamy at Georgia
Institute of Technology, J. Jakowski at the University of Tennessee Knoxville, J. Oelgoetz at Austin
Peay State University, and S. Khondaker at the University of Central Florida, and J. Wilgenbusch at
the Minnesota Supercomputing Institute at the University of Minnesota for helpful discussions and
support. This work used the Extreme Science and Engineering Discovery Environment (XSEDE),
which is supported by National Science Foundation grant number ACI-1053575. This research used
resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User
Facility supported under Contract DE-AC05-00OR22725.
19
References
[1] R.R. Hartmann, J. Kono, and M.E. Portnoi. Terahertz science and technology of carbon
nanomaterials. Nanotechnology 25, page 322001, 2014.
[2] F. Bonacorso. Graphene photonics and optoelectronics. Nat. Photonics, 6(749), 2010.
[3] J.C. Charlier, X. Blase, and S. Roche. Electronic and transport properties of carbon nanotubes.
Rev. Mod. Phys., 79:677, 2009.
[4] A.H. Castro Neto, F. Guinea, N.M.R. Peres, K.S. Novoselov, and A.K. Geim. The electronic
properties of graphene. Rev. Mod. Phys., 81:109, 2009.
[5] H. Suzuura and T. Ando. Phonons and electron-phonon scattering in carbon nanotubes. Phys.
Rev., B65:235412, 2002.
[6] J.-Y. Park, S. Rosenblatt, Y. Yaish, V. Sazonova, H. Ustunel, S. Braig, T. A. Arias, Piet W.
Brouwer, and P.L. McEuen. Electron-Phonon Scattering in Metallic Single-Walled Carbon
Nanotubes. Nano Lett. 4, 4(517), 2004.
[7] S. Iijima. Helical microtubules of graphitic carbon. Nature (London), 354:56, 1991.
[8] W. Choi, S. Hong, J.T. Abrahamson, J.-H. Han, C. Song, N. Nair, S. Baik, and M. S. Strano.
Chemically driven carbon-nanotube-guided thermopower waves. Nature Mater., 9:423, 2010.
[9] S. Motavas, B. Omrane, and C. Papadopoulos. Large area patterning of carbon nanotube ring
arrays. Langmuir, 25(8):4655, 2009.
[10] G.N. Afanasiev and V.M. Dubovik. Some remarkable charge-current configurations. Phys.
Part. Nucl., 29:366, 1998.
[11] G.N. Afanasiev. Simplest sources of electromagnetic fields as a tool for testing the reciprocity-
like theorems. J. Phys., D34:539, 2001.
[12] A.D. Boardman, K. Marinov, N. Zheludev, and V.A. Fedotov. Dispersion properties of non-
radiating configurations: Finite-difference time-domain modeling. Phys. Rev., E72:036603,
2005.
[13] N. Papasimakis, V.A. Fedotov, K. Marinov, and N.I. Zheludev. Gyrotropy of a metamolecule:
wire on a torus. Phys. Rev. Lett., 103:093901, 2009.
[14] T. Kaelberer, V.A. Fedotov, N. Papasimakis, D. P. Tsai, and N.I. Zheludev. Toroidal dipolar
response in a metamaterial. Science, 330:1510, 2010.
[15] L. Liu, G.Y. Guo, C.S. Jayanthi, and S. Y. Wu. Colossal paramagnetic moments in metallic
carbon nanotori. Phys. Rev. Lett., 88:217206, 2002.
20
[16] K. Sasaki and Y. Kawazoe. Characteristic behavior of toroidal carbon nanotubes: Kinematics
of persistent currents. Prog. Theor. Phys, 112(3):369, 2004.
[17] M. Encinosa and M. Jack. Elliptical tori in a constant magnetic field. Physica Scripta, 73:439,
2006.
[18] M. Encinosa and M. Jack. Dipole and solenoidal magnetic moments of electronic surface
currents on toroidal nanostructures. Journal of Computer-Aided Materials Design, 14(1):65,
2007.
[19] J. Williamson and M. Encinosa. Quantum toroidal moments of an elliptic toroidal helix in a
constant magnetic field. quant-ph/1108.5097.
[20] M. Encinosa and J. Williamson. Toroidal moments of schrodinger eigenstates.
quant-ph/1106.4248v1.
[21] F. H. L. Koppens et al. Graphene plasmonics: A platform for strong light-matter inter-actions.
Nano Lett. 11, 11(3370), 2011.
[22] G. Bryant. Hubbard model for intermediate dimensional excitons. Phys. Rev., B49(23):16129,
1994.
[23] V.N. Popov and L. Henrard. Comparative study of the optical properties of single-walled
carbon nanotubes within orthogonal and nonorthogonal tight-binding models. Phys. Rev.,
B70:115407, 2004.
[24] I.V. Bondarev, L.M. Woods, and K. Tatur. Strong exciton-plasmon coupling in semiconducting
carbon nanotubes. Phys. Rev., B80:085407, 2009.
[25] M.J. Leamy and A. DiCarlo. Phonon spectra prediction in carbon nanotubes using a manifold-
based continuum finite element approach. Computer Methods in Applied Mechanics and En-
gineering, 198(17-20):1572, 2009.
[26] M.J. Leamy. Calculation of phonon dispersion in carbon nanotubes using a continuum-
atomistic finite element approach. AIP Advances, 1:041702, 2011.
[27] S. Balay, J. Brown, K. Buschelman, V. Eijkhout, W. Gropp, D. Kaushik, M. Knepley, L. Curf-
man McInnes, B. Smith, and H. Zhang. Petsc users manual. PETSc 4.3, ANL-95/11, October
15, 2013. http://www.mcs.anl.gov/petsc.
[28] G. Klimeck, S.S. Ahmed, H. Bae, N. Kharche, and R. Rahman. Atomistic simulation of
realistically sized nanodevices using NEMO 3-d: Part i - models and benchmarks. Special Issue
on Nanoelectronic Device Modeling in IEEE Transactions on Electron Devices, 54(9):2079,
2007.
21
[29] G. Klimeck, S.S. Ahmed, N. Kharche, M. Korkusinki, and M. Usman. Atomistic simulation
of realistically sized nanodevices using NEMO 3-d: Part II - applications, special issue on
nanoelectronic device modeling. IEEE Transactions on Electron Devices, 5(9):2090, 2007.
[30] B.P. Haley, S. Lee, M. Luisier, H. Ryu, F. Saied, S. Clark, H. Bae, G. Klimeck, and F. Saied.
Advancing nanoelectronic device modeling through peta-scale computing and deployment on
nanoHUB. J. Phys. : Conf. Ser, 180:012075, 2009.
[31] M. Luisier and G. Klimeck. Atomistic full-band simulations of silicon nanowire transistors:
Effects of electron-phonon scattering. Phys. Rev., B94(22):155430, 2009.
[32] M. Luisier and G. Klimeck. Performance analysis of statistical samples of graphene nanoribbon
tunneling transistors with line edge roughness. Appl. Phys. Lett., 94(22):223505, 2009.
[33] S. Kim, M. Luisier, T.B. Boykin, and G. Klimeck. Computational study of heterojunction
graphene nanoribbon tunneling transistors with p-d orbital tight-binding method. Appl. Phys.
Lett., 104(24):224113, 2014.
[34] B. Aradi, B. Hourahine, and Th. Frauenheim. DFTB+, a sparse matrix-based implementation
of the DFTB method. J. Phys. Chem., A111(26):5678, 2007.
[35] G. Zheng, M. Lundberg, J. Jakowski, T. Vreven, M.J. Frisch, and K. Morokuma. Implemen-
tation and benchmark tests of the DFTB method and its application in the ONIOM method.
Int. J. Quant. Chem., 109(9):1842, 2009.
[36] S. Garashchuk, J. Jakowski, L. Wang, and B.G. Sumpter. Quantum trajectory-electronic
structure approach for exploring nuclear effects in the dynamics of nanomaterials. J. Chem.
Theory Comput., 9:5221, 2013.
[37] L. Wang, J. Jakowski, and S. Garashchuk. Adsorption of a hydrogen atom on a graphene
flake examined with quantum trajectory/electronic structure dynamics. J. Phys. Chem.,
C118:16175, 2014.
[38] V. Perebeinos, J. Tersoff, and P. Avouris. Effect of exciton-phonon coupling in the calculated
optical absorption of carbon nanotubes. Phys. Rev. Lett., 94:027402, 2005.
[39] L.M. Woods and G.D. Mahan. Electron-phonon effects in graphene and armchair (10, 10)
single-wall carbon nanotubes. Phys. Rev., B61(16):10651, 2000.
[40] L.V. Keldysh. Diagram technique for nonequilibrium processes. Sov. Phys. JETP, 20(4):1018,
1965.
[41] A.P. Jauho, N.S. Wingreen, and Y. Meir. Time-dependent transport in interacting and non-
interacting resonant-tunneling systems. Phys. Rev., B50:5528, 1994.
[42] S. Datta. Electronic Transport in Mesoscopic Systems. Cambridge University Press (NY),
1995.
22
[43] M.P. Anantram and T.R. Govindan. Conductance of carbon nanotubes with disorder: A
numerical study. Phys. Rev., B58(8):4882, 1998.
[44] M. Encinosa and M. Jack. Quantum electron transport in toroidal carbon nanotubes with
metallic leads. Journal of Molecular Simulations, 34(1):9, 2008.
[45] M. Encinosa and M. Jack. Excitation of surface dipole and solenoidal modes on toroidal
structures. physics/0604214.
[46] I.V. Bondarev, K. Tatur, and L.M. Woods. Surface exciton-plasmons and optical response of
small-diameter carbon nanotubes. Opt. and Spectr, 108:376, 2010.
[47] P.R. Amestoy, I.S. Duff, J. Koster, and J.-Y. L’Excellent. A fully asynchronous multifrontal
solver using distributed dynamic scheduling. SIAM Journal of Matrix Analysis and Applica-
tions, 23(1):15, 2001.
[48] P.R. Amestoy, A. Guermouche, J.-Y. L’Excellent, and S. Pralet. Hybrid scheduling for the
parallel solution of linear systems. Parallel Computing, 32(2):136, 2006.
[49] E. Agullo, J. Demmel, J. Dongarra, B. Hadri, J. Kurzak, J. Langou, H. Ltaief, P. Luszczek, and
S. Tomov. Numerical linear algebra on emerging architectures: The PLASMA and MAGMA
projects. J. Phys. : Conf. Ser., 180(1):012037, 2009.
[50] J. Dongarra, M. Gates, A. Haidar, Y. Jia, K. Kabir, P. Luszczek, and S. Tomov. Portable
HPC Programming on Intel Many-Integrated-Core Hardware with MAGMA Port to Xeon Phi.
PPAM 2013, Warsaw, Poland, Sep 2013.
[51] K.R. Dienes and B. Thomas. Isospectral But Physically Distinct: Modular Symmetries and
their Implications for Carbon Nanotori. Phys. Rev., B84:085444, 2011.
[52] A. Kuzmin, M. Luisier, and O. Schenk. Fast Methods for Computing Selected Elements of
the Green’s Function in Massively Parallel Nanoelectronic Device Simulations. Euro-Par 2013,
LNCS 8097, F. Wolf and B. Mohr and D. an Ney (Eds.), Springer-Verlag Berlin Heidelberg,
2013.
[53] C. Petra, O. Schenk, M. Lubin, and K. Gartner. An augmented incomplete factorization ap-
proach for computing the Schur complement in stochastic optimization. SIAM J. Sci. Comput,
36(2):C139, 2014.
23
Recommended