QRing − a scaleable, parallel software tool for quantum transport

simulations in carbon nanodevices using linear solvers on Intel

Xeon/Phi and GPU architectures

Mark A. Jack∗1, Benjamin Prather†2, Adam Byrd1, Leon W. Durivage3, and Mario R.Encinosa1

1Florida A&M University, Department of Physics, Tallahassee, FL 32307.2Arizona State University, Department of Physics, Tempe, AZ 85281.

3Winona State University, Department of Physics, Winona, MN 55987.

June 4, 2015

Abstract

The ability of a nanomaterial to conduct charge is essential for many nanodevice applica-tions. Numerous studies have shown that disorder including defects and phononic or plasmoniceffects can disrupt or block electric current in nanomaterials. An accurate theoretical account ofboth electron-phonon and electron-plasmon coupling in carbon nanotube- and nanoring-basedstructures is key in order to properly predict the performance of these new nanodevices. Amulti-scale continuum approach developed by collaborator M.J. Leamy (Georgia Institute ofTechnology) forms the basis for reduced-order phonon modeling of carbon nanostructures. Thedevice Green’s function Gd(E), derived within a non-equilibrium Green’s function formalism(NEGF) will function as key input to determine all transport observables. The transmissionfunction T (E) may be calculated from Gd(E) by inverting the Hamiltonian matrix at each en-ergy step E. Integrated observables such as for example source-drain current ISD may then becalculated by integrating over T (E). Using the parallel sparse matrix library PETSc providestools to optimally divide memory and computational tasks required in inverting the Hamil-tonian matrix across multiple compute cores or nodes. The results at different energy levelsare integrated via a second layer of parallelism to obtain integrated observables. Because theinversion at each energy step dominates the run time and memory use of the code, it is im-portant that the code scales well to realistic system sizes. Several different direct and iterativelinear solvers and libraries were compared for their scalability and efficiency, including the In-tel MKL shared-memory, direct, dense solver, the MUltifrontal Massively Parallel direct Solver(MUMPS), and iterative, sparse solvers bundled with PETSc. The National Science Founda-tion’s Extreme Science and Engineering Discovery Environment (XSEDE) resource Stampede atthe Texas Advanced Computing Center as well as regional resources in Florida provided by theSunshine State Education and Research Computing Alliance (SSERCA) are used for develop-ment and benchmarking, while the Department of Energy’s Oak Ridge Leadership ComputingFacilities (DoE OLCF) like Titan and NICS/Beacon are used for physics production runs.

∗mark.jack@famu.edu†Benjamin.Prather@asu.edu

1

Introduction

The reliable electrical and magnetic manipulation of information and energy in nanostructures at

the 1- to 100-nm scale is crucial for the development of novel, nanoelectronic devices [1]. Such ma-

nipulation must be highly precise, which requires understanding such phenomena as Shottky barri-

ers at metallic contacts, dissipation, material defects, and contact with the substrate. Carbon-based

nanostructures promise to be ideally suited for inexpensive, fast and low-power next-generation

electronics (nanoelectronics) [2, 3, 4]. Practical predictions of device behavior must account for

electrical resistance due to the presence of phonons and due to the electrons’ inherent excitation of

phonons [5, 6]. Graphene-based allotropes such as toroidal carbon nanotubes [7] hold the promise

of completely new nanodevice [3, 4, 5, 6, 8] and metamaterials applications [9, 10, 11, 12, 13, 14]. In

addition to persistent current and Aharonov-Bohm effects [15, 16], new electromagnetic moments

such as the toroidal moment will be generated by ring currents in a carbon nanotorus driven by a

voltage between two metallic leads [17, 18, 19, 20]. An example of the device geometry is sketched in

Fig. 1 including a tightbinding-derived model for the magnetic flux dependence of the source-drain

current (b). Another quickly expanding area of nanodevice applications is nanoplasmonics [21],

Fig. 1: a) Sketch of nanotorus resting on two metallic leads. b) Aharonov-Bohm oscillations ofcurrent vs. flux in a magnetoresistance plot.

where coherent electronic oscillations, plasmons, can be tuned via gate voltage and doping. In a

metamaterial of nanorings for example the interference of ring-generated electromagnetic radiation

from electronic surface currents driven by an incoming polarized wave front can lead to new opti-

cal response characteristics [9, 10, 11, 12, 13, 14]. Electronic correlation and exchange effects on

quantum transport, including exciton-phonon and exciton-plasmon coupling, are described effec-

tively in a tightbinding transport description using a non-equilibrium Green’s function formalism.

A multi-scale analysis tool was developed, which (1) rigorously quantifies these interactions, and

(2) yields open source tools available to the research community which allow general geometries

to be analyzed for inelastic effects in charge transport. Transport observables like transmissivities,

2

integrated currents, and magnetoresistance may be calculated via NEGF for realistic system sizes.

Transmissivities, integrated currents, and magnetoresistance might be sensitive to particular, sup-

pressed vibrational modes. The utilized model will permit the expected temperature-dependence

of resistance/conductivity and power loss to be expressed in the technically very important long

phonon-wavelength regime in nanoelectronics. Having in mind applications in the very new area

of nanoplasmonics on graphene-based architectures [21], electronic correlation and exchange effects

on quantum transport including electron-plasmon coupling can be described in a Hubbard model

[22, 23, 24] that allows a generalization of tightbinding transport calculations using NEGF. The

use of fast, parallel, sparse-matrix libraries will allow expansion to beyond tightbinding accuracy.

Transport observables may then be calculated quickly and efficiently for realistic system sizes of

10,000 atoms and more.

A curved manifold-based approach is employed in the electronic structure, phonon disper-

sion, and electron-phonon interaction modeling [25, 26]. In a collaboration with Georgia Tech’s

M.J. Leamy and Florida A&M University’s M. Encinosa the first two, decoupled models have al-

ready been completed using closely-related techniques for carbon nanotubes and nanotoroids. A

recently developed multi-scale continuum approach (M.J. Leamy) forms the basis for reduced-order

phonon modeling of carbon nanostructures. Electron-plasmon and exciton-plasmon couplings in

carrier transport may be incorporated as well in an effective quasi-particle tightbinding description

after appropriate transformation (Boguliobov canonical transformation) [24] and diagonalization of

the Hamiltonian containing a free exciton, a free plasmon and an exciton-plasmon interaction term.

The C++ code has an object-oriented and modular structure using MPI parallelism and sparse

matrix software libraries such as PETSc. PETSc includes routines to manipulate large sparse

matrices quickly and efficiently using an optimal distribution of the memory allocation for the

matrix over multiple cores. The algorithm is setup in such a way that the transmission function

T (E) may be calculated at a single energy E by inverting the Hamiltonian matrix to obtain the

transport Green’s function via the use of PETSc on multiple compute cores. Integrated observables

such as the source-drain current ISD for example can then be calculated by integrating over the

transmission function values for a pre-defined energy range using a second layer of parallelism

using MPI. High-performance computing resources are available via the new Florida-wide HPC

network SSERCA.org (Sunshine State Education and Research Computing Alliance), primarily

for code development and benchmarking, while large-scale simulation runs using 1000 and more

cores are conducted on national computational resources such as the National Science Foundation

XSEDE’s Stampede cluster at the Texas Advanced Computing Center (TACC) and the Department

of Energy’s Oak Ridge National Laboratory’s Leadership Computing Facilities (OLCF) Titan and

NICS/Beacon.

The research team of M. Jack and a number of summer students supported via NSF XSEDE

and BlueWaters/Shodor programs have developed a fast and versatile parallel software tool, QRing,

setout to estimate quantum scattering corrections in quantum charge transport in carbon nanos-

tructure devices effectively for realistic nanosystems modeling of up to 30,000 atoms using a tight-

binding, non-equilibrium Green’s function formalism. The code allows an effective and fast treat-

3

ment of first-order electron-phonon scattering corrections in single-electron transport for nanoscale

transistors based on straight or toroidal carbon nanotubes including static external magnetic fields

under a small bias potential for realistic device dimensions, which cannot be obtained with typical

ab-initio techniques such as density functional theory. A multi-scale continuum approach developed

by collaborator M.J. Leamy at Georgia Institute of Technology’s Department of Mechanical En-

gineering forms the basis for reduced-order phonon modeling of carbon nanostructures. Arbitrary

low-energy phonon modes important for nanodevice performance can be modeled and are read into

the code to incorporate electron-phonon scattering corrections.

The code has a modular organization, programmed in C++, using an object-oriented approach

to store the effective Hamiltonian for the device region as a large, sparse, complex and banded

matrix over the shared memory of individual nodes. The use of built-in MPI parallelism and a

state-of-the-art high-performance computing library, PETSc, solves for the electronic Green’s func-

tion of the system via fast, parallel inversion of the Hamiltonian matrix with which any transport

observable or band-structure characteristic can be computed. Using the parallel sparse matrix

library PETSc [27], PETSc provides tools to optimally divide memory and computational tasks

required in inverting the Hamiltonian matrix across multiple compute cores or nodes. Thus, mul-

tiple different nanodevice settings such as graphene nanoribbons, carbon nanotubes with ’kinks’

or non-closing nanorings can be just as easily implemented as lattice defects, impurities, and sim-

ilar scattering effects by a simple re-definition of the Hamiltonian matrix in the code. Equally,

new quasi-particle effects such as excitons and plasmons with exciton-plasmon coupling can be

investigated with an adjusted device Hamiltonian in a Hubbard model and effective tightbinding

description. Because the inversion at each energy step dominates the run time and memory use of

the code, it is important that the code scales well to realistic system sizes.

Our simulation code QRing can be considered a complimentary software tool in the space of

nanoscale device and quantum transport simulations. In this context, state-of-the-art programs

such as the tools OMEN and NEMO 3D created at Purdue University with petascale simulation

capabilities come to mind [28, 29, 30, 31, 32, 33]. In particular, OMEN provides modeling capability

for transport primarily in nanowires of III-V semiconductor materials in form of a multi-level

parallelism code for typical cross sections, gate lengths and body thicknesses of a few 10ths of

nanometers [28, 29, 30, 31]. Some recent work has also been conducted for graphene nanoribbons

[32, 33]. Purdue’s simulation tool can perform typical simulations in few minutes on up to roughly

200,000 compute cores. In comparison, our tool QRing focuses on carbon nanotube- and graphene-

based devices with comparable tube and ring widths of a few nm and tube lengths or torus diameters

of 40− 80 nm. QRing has been tested and shows excellent scaleability on up 16,000 compute cores

for system sizes of at least 24,000 atoms with simulation runs conducted on Titan at OLCF in

under one hour. QRing has been tested on large-scale computer clusters provided by XSEDE and

OLCF and shows excellent scaleability for system sizes of up to 30,000 atoms with. There is no

restriction in speed-up by simply increasing the number of available cores. The only restrictions to

occur are due to having to limit simulations to less than roughly 30,000 atoms due to maximally

available shared memory per compute core in order to avoid increased latencies from inter-node

4

communications.

In addition, OMEN models phononic corrections in a self-consistent tighbinding and NEGF

formalism with a Poisson solver but does not directly contain a module to treat other quasi-

particle corrections such as exciton-plasmon coupling, a core feature to be integrated with the

existing description for electron-phonon coupling in QRing as described further in this proposal.

Matrix elements for multi-electron scattering and correlation effects are currently being determined

by collaborators J. Jakowski at University of Tennessee Knoxville and J. Oelgoetz at Austin Peay

State University using the new technique density-functional tightbinding [34, 35, 36, 37] and will

be included in QRing in the next few months before the suggested proposal start date of July

1, 2015, thus providing a similar level of precision of theoretical prediction as tools like OMEN

but with additional quantum corrections. Due to its modular structure and reliance on state-

of-the-art libraries such as PETSc as described above these effects may be incorporated easily

in QRing by a redefinition of the Hamiltonian in tightbinding approximation. Also higher-order,

beyond tightbinding calculations can be done by appropriately defining additional off-diagonal

matrix elements with matrix calculations handed off to PETSc’s linear solver algorithms. QRing

simulation runs will be compared to existing thermoelectric experimental data on carbon nanotubes

provided by S. Khondaker’s group at the University of Central Florida’s NanoScience Technology

Center. Semi-empirical input parameters such as an electronic hopping term can be fine-tuned

to e.g. adjust for resistive effects at the contact sites between device region and metallic leads,

currently not treated theoretically at an atomistic level in the code, to more accurately estimate

the experimental results.

1 Theory and computational methodology

A research collaboration composed of faculty at Georgia Institute of Technology (M.J. Leamy)

and Florida A&M University (M. Jack) has developed two, decoupled models to describe phonons

on the one hand and electron transport on the other using closely related techniques for carbon

nanotubes and nanotoroids. A recently developed multi-scale continuum approach by Leamy [25,

26] forms the basis for reduced-order phonon modeling of carbon nanostructures (see Fig. 2). This

approach employs intrinsic basis vectors defined on a reduced-dimension surface (manifold) of the

nanostructured material. Changes in interatomic potential energy arising from lattice vibrations

are equated to changes in continuum strain energy, allowing non-quantum atomistic behavior to

be captured using continuum techniques. A subsequent finite element discretization results in a

significant decrease in the deformation degrees of freedom present, while at the same time accurately

capturing the acoustic range of the phonon spectra. Results for phonon modes show very good

agreement between the multi-scale approach and experimental results and ab-initio calculations

[38, 39].

5

Fig. 2: a) Continuum computation approach used to study reduced-dimension materials. b) Typicalphonon mode predicted using the continuum approach.

1.1 Non-equilibrium Green’s function method (NEGF)

All observables can be derived from the (advanced / retarded) Green’s function G(a,r)d (k) of the

device region [40, 41, 42]: [E(k)I −H(k)− ΣL − ΣR ± iη

]G

(a,r)d (k) = I. (1)

The effects of the left (right) metallic lead are folded into an effective Hamiltonian shown in Eq. 1

in form of self-energy corrections ΣL,R with H(k) describing the device region. I symbolizes the

unit matrix. In tightbinding approximation, only the overlap integrals of the nearest-neighbor pz-

orbitals are considered and are described in terms of a general electronic ’hopping’ parameter te

for transmission along the hexagonal lattice structure. With the determined Green’s function, the

current ISD can be calculated with the transmission function (transmissivity) T (E) via:

T (E) = Trace[ΓLG

rΓRGa], (2)

ISD =2e

~

∫dE T (E)

[fL(E)− fR(E)

]. (3)

with fL,R(E) as Fermi distributions at the two contacts and ΓL,R as nanotube-to-lead couplings

(leads are held at thermodynamic equilibrium while transport in the device region is out-of-

equilibrium). One can calculate density-of-states D(E) and transmission function T (E) for electron

transport through the torus between two attached metallic leads under a small voltage bias.

The effective Hamiltonian yields a sparse matrix with the non-zero elements mostly arranged in

a banded structure along the diagonal and at the corners of the matrix for torus closure [43, 44, 45].

To be exact, for example for a (10, 10) armchair or a (10, 0) zigzag chirality, due to the nanoring

(or nanotube) geometry with ring-to-ring coupling, a self-similar organization of the Hamiltonian

matrix occurs into a subset of, for example, 40×40 or 20×20 banded, sparse matrices, respectively.

6

1.2 Phonon-modulated electronic hopping term, electron-phonon coupling andphonon-modulated electron-electron correlation

The interaction of electrons with phonons in charge transport will be modeled using additional

terms in the Hamiltonian, which account first for a phonon-modulated electronic hopping term

te. We follow the theoretical description provided in [38, 39]. Low-energy phonon modes for

carbon nanorings of arbitrary size and chirality are calculated in a continuum model description

via a finite-difference-time-domain code (FDTD), provided by collaborator M.J. Leamy at Georgia

Institute of Technology’s Mechanical Engineering Department. Amplitudes of oscillations of atomic

coordinates for different low-energy phonon modes including breathing, stretching and torsional

modes are now included in our tightbinding transport calculations of transport observables such

as transmissivities, source-drain currents etc. The modified strength of the term te will be derived

from the lattice deformation accompanying each phonon mode and from the occupancy of the

phonon modes at the temperature of the system [25, 26]: First, an effective deformation potential

of the form V1 = g1

(uxx + uyy

);uxx = ∂ux

∂x + uzR ;uyy =

∂uy∂y will change the effective electronic

hopping term contained in the diagonal and off-diagonal matrix elements of the Hamiltonian.

ux,y,z are the local deformations of the torus in the continuum model; R is the the torus’ large

curvature radius. Parameter g1 is given as approximately 30 eV. The corresponding Hamiltonian

matrix elements are given in the literature [39]. The effect of phononic corrections may now be

captured in the Hamiltonian by superimposing a hexagonal lattice structure of carbon atoms onto

the deformed torus surface and replacing the electronic hopping parameter te,h = −3.1 eV by an

effective parameter te,hij due to the relative displacement of adjacent carbon atoms i, j during lattice

vibrations characterized quantum mechanically by phonon modes. In good first approximation,

te(h) is now enhanced by a term proportional to the atomic displacements δij = r′ij − rij and scales

linearly with the factor ε = 5.4 eV A to be derived in an atomistic calculation e.g. via a Morse

potential approach for the overlap of pz-orbitals of adjacent C atoms [38, 39]:

te(h)ij = te(h) + ε ηij δij with:

ηij =

(r′ij−rij

)∥∥∥r′ij−rij∥∥∥ ; δij = δi − δj ;

δij = r′ij − rij .

(4)

For a more advanced treatment of vibrational modes, following the description in [38, 39], the

Hamiltonian will also contain a free-phonon term and, more importantly, a term describing electron-

electron interactions, which will now be modulated by the displacements of the atomic cores. In

the rigid-ion approximation, valid for long-wavelength and thus low-energy phonons, the atomic

cores move adiabatically slowly to each atom’s electronic charge distribution so that the electron

distribution is modeled to instantaneously adjust to the position of the positive ion cores. Thus,

the oscillatory displacement of the atomic cores will lead to an effective oscillatory separation of the

charge clouds of neighboring atomic cores and thus to a change of the effective Coulomb repulsion

between different electrons. This effect can be treated by additional off-diagonal contributions V2

7

to the Hamiltonian matrix,

V2 = g2 ei3η(uxx − uyy + i2uxy

); 2uxy =

∂ux∂y

+∂uy∂x

; g2 ≈ 1.5 eV, (5)

and can be parameterized in form of two terms − a linear electron-phonon coupling term and

a phonon-modulated electron-electron correlation term. The matrix element for linear electron-

phonon coupling e.g. in a (10, 10) armchair carbon nanoring is listed here to include low-energy

(acoustic) phonons (small Q):

Mqz ,Qz = −4iMqz

(qηQ

)XQz ; Mq = νqρ

2e(q). (6)

The parameter g2 measures the relative strength of this term with respect to the deformation

potential contribution proportional to g1. νq and ρe(q) are the electron velocity and electron

density, respectively, at electron momentum q. Phonon frequencies ωph and phonon polarization

vectors η can all be directly extracted for each atomic lattice site from the finite-element based

simulation by properly discretizing the continuum-model description of the ring distortions in the

finite-element based simulation. For a given vibrational mode, this information will then be read

into the electronic transport Hamiltonian for each matrix element.

1.3 Inclusion of multi-electron correlations via density functional tightbinding

In collaboration with the University of Tennessee Knoxville and Austin Peay State University, re-

searchers J. Jakowski and J. Oelgoetz will provide the necessary matrix elements for electron-phonon

coupling and phonon-modulated electron-electron interaction via a new methodology called den-

sity functional tightbinding (DFTB) [34]. In DFTB, both electronic and a subset of (light) nuclear

coordinates may be treated quantum mechanically with an exact, DFT-like treatment of valence

electrons while wave functions of the lighter nuclear coordinates are estimated approximately with a

reduced basis set expansion with the heavier nuclei treated classically [35, 36, 37]. A self-consistent

treatment of electron density in DFTB can be used routinely in dynamics of roughly 1000 atoms.

For calculational simplicity and efficiency, a non-self-consistent treatment will be used for larger

systems from which the device Green’s function is evaluated at each energy. A self-consistent de-

scription may be included at a later stage. While a full MD-type calculation for torus sizes of

several 30,000 atoms and more is not possible in DFTB, the plan is to calculate the corrections for

a short segment of e.g. a (10, 10) torus chirality consisting of a few connected small-diameter rings

of a 1000+ atoms and then to scale these to the full torus dimensions using symmetry arguments

of point groups of a graphene lattice compactified to a toroidal surface with chiral indices (m,n).

Jakowski possesses a dual appointment as researcher with ORNL and as XSEDE staff member

with computational resources readily available to both him and Oelgoetz at ORNL (Titan), NICS

(NICS/Beacon) and XSEDE (Stampede, Darter). This collaborative activity is conducted as an

XSEDE Extended Collaborative Support Service (ECSS) with Jakowski acting as XSEDE staff

support. A list of tasks to be completed is sketched below:

• QRing code ported to NSF XSEDE resources;

8

• Interface between standalone QRing and DFTB programs;

• Validation of QRing/DFTB approach for quantum transport;

• Analysis of performance and optimization of QRing/DFTB;

• Integration of DFTB and QRing into a single binary.

An expansion of this research program to include other scattering corrections such as lattice

defects, adsorbed atoms, etc. via DFTB is planned. The overall model would allow to generally

compute the expected temperature-dependence of resistance/conductivity and power loss in the

technically very important long-phonon-wavelength regime, which prototypical carbon nanostruc-

tures will experience in next-generation electronics. Results from the model will be compared

with experimental data made available by S. Khondaker at the University of Central Florida’s

NanoScience Technology Center.

1.4 Experimentally realistic metallic lead geometry

Tight-binding transport calculations typically rely on a one-to-one identification between lattice

sites comprising the contact layer of the metallic lead to the carbon atoms of the device region.

A major obstacle that must be overcome when a toroidal nanotube is placed atop a rectangular

semi-infinite lead (see Fig. 1) is that the alignment between metallic and carbon sites is no longer

maintained. The locations of the carbon atoms comprising the tube/torus trace an arc with some

carbon atoms in direct contact with the lead while the lattice sites of the metallic lead are generally

taken to comprise a rectangular lattice. The straightforward calculation of the Green’s function

G(r, r′

)for transport along one semi-infinite metallic lead yields 1

G(r, r′

)= − 16me

π~2LyLz

∑m,n

sin(mπyLy

)sin(mπy′Ly

)sin(nπzLz

)sin(nπz′Lz

)

·+∞∫−∞

dksin kx sin kx′(

k + kmn + iη)(k − kmn − iη

) (7)

with k, kmn trivially related to the energy E and particle in a box energy eigenvalues Emn through

kmn =

√2me

(EFermi − Emn

)/~ (8)

In one approach, in order to obtain the Green’s function values, the longitudinal degree of

freedom along the leads is treated as discrete, and after binning, the coordinates of the carbon

atoms in contact with a metallic lead are searched which are closest to metallic lattice sites. The

metallic Green’s function value may then be computed at that site as an approximation for use

in the device Hamiltonian. As this approach can be criticized as being time-inefficient and would

artificially introduce a resolution of accuracy into the calculation that is unwarranted and arguably

1This configuration corresponds to the original, computationally simpler implementation with the metallic leadstouching the torus region laterally on both sides in a few atomic contact sites.

9

outside of the scope of the method, an alternate approach was adopted: It is well known that the

expression listed above produces a Green’s function that is infinite when x = x′ which map into

diagonal elements of the matrix for the device region, Gd. The goal was to produce a hybrid method,

one that keeps the longitudinal degrees of freedom continuous to allow for a trivial matching to

carbon sites, but remains finite when x = x′. This is accomplished by introducing integration limits

that correspond to the cutoff momentum kcut of a lattice of spacing a , i.e. kcut = πa . The integral

can then be evaluated in very good approximation by contour methods to yield the result

Gcut

(r, r′

)= − 8me

~2LyLz

∑m,n

sin(mπyLy

)sin(mπy′Ly

)sin(nπzLz

)sin(nπz′Lz

)

·exp

(ikmn x>

)sin(kmn x<

)kmn

(9)

Only poles that fall within the cutoff kcut = πa contribute to this Green’s function. The argument of

the exponential becomes imaginary when Emn > EFermi, which allows for a controllable convergence

criterion. For typical values of EFermi and Emn, suitable convergence is obtained with on the order

of 1000 summation terms for each index m,n if the lead has a square profile, or less, should the

lead be flatter along the z-direction.

1.5 Exciton-plasmon coupling as example of advanced quasi-particle corrections

The implementation of a new, upgraded Hamiltonian into the existing code to include plasmonic ef-

fects e.g. in form of electron-plasmon coupling terms and eventually expanded to an exciton-plasmon

is also envisioned in the near future. This will entail a significant reorganization and expansion of

the part of the code that defines the distributed Hamiltonian matrix, which is anticipated to take a

few months with the necessary testing and debugging phase of the code. A schematic of how a new

tightbinding Hamiltonian H for strong exciton-surface-plasmon coupling (as demonstrated in semi-

conducting and metallic carbon nanotubes) can be constructed via a quasi-particle description is

illustrated below with H = Hplasmon+Hexc+Hint and Hplasmon, Hexc, and Hint as the free-plasmon,

free-exciton and interaction terms, respectively [22, 23, 46, 24]:

Exciton-plasmon interaction:

Hint = − e

mec

∑n

A(n)[

pn −e

2cA(n)]

+∑n

d(n)∇nφn. (10)

Boguliobov canonical transformation (diagonalization for quasi-particle description):

Hint =∑

k,µ=1,2

~ωµ(k)· (k)ξµ

(k)

+ E0. (11)

2 History and early benchmarks on TACC/Ranger cluster

Historically, a much simplified version of the code existed as a serial Fortran program developed by

co-investigator M. Encinosa at the NASA Ames Research Center in the summers 2000 and 2001

10

to study charge transport in armchair carbon nanotubes. A fast recursive algorithm to calculate

the transport Green’s function for system sizes of up to 4,000 atoms had been implemented and

was extended to include armchair nanoring devices for initial physics studies of transport under

magnetic flux. Typical calculations of ISD curves and magnetoresistance plots (ISD vs. B) for

(3, 3) armchair carbon nanorings with 3600 atoms (20 nm diameter) completed in about 25 hours

on a single processor of a computer work station. With the need to be able to model realistic

device geometries with nanorings of up to 200 to 500 nm in diameter, it was decided that a faster

and more modular parallel algorithm would be necessary. A project emerged to completely re-

write the simulation in form of a newly reorganized C++ program with an object-oriented and

modular structure. An object-oriented C++ code with MPI parallelism and the use of sparse

matrix software libraries such as PETSc [27] was developed by M. Jack and summer students L.W.

Durivage (Winona State U.) and A. Byrd (Florida A&M U.) in summers 2010 and 2011 as part

of the BlueWaters/Shodor Undergraduate Petascale Education Program (UPEP). PETSc contains

a range of routines in C++ using a strongly modular and object-oriented approach to handle

large linear systems. The PETSc library itself, and through the use of add-on packages such as

MUMPS [47, 48] includes routines to calculate large sparse matrices quickly and efficiently using

an optimal distribution of the memory allocation for the matrix over multiple cores. This modular

approach was also used in the organization of the transport code itself together with PETSc’s

parallel inversion routines to be able to later include more advanced transport modeling beyond

tightbinding and for excitonic and/or plasmonic scattering corrections via simple replacement of

the effective Hamiltonian [22, 23, 46, 24].

The algorithm is setup in such a way that the transmission function T (E) may be calculated at

a single energy E by inverting the Hamiltonian matrix to obtain the transport Green’s function via

the use of PETSc on multiple compute cores. The source-drain current ISD for example can then

be calculated by integrating over the transmission function values for a pre-defined energy range

(depending on the source-drain voltage), see section II. For an accurate estimate of integrated ob-

servables such as ISD with up to 3 decimal values numerical accuracy, T (E) values are calculated

for a grid of roughly 8,000 energy values E between −0.2 and 0.2 eV. These 8,000 values are cal-

culated via a second layer of parallelism using MPI in an embarrassingly parallel fashion. Through

increase of the number of cores for each distributed PETSc/MUMPS [47, 48] matrix inversion also

the overhead in communication between cores will increase, slowing down the overall calculation.

Thus, the number of cores necessary to do a single matrix inversion should be kept to a minimum

depending on the memory requirements to store that matrix. If the total processor number is not

an integer multiple of the number of cores in each group handling a single PETSc/MUMPS matrix

inversion at a single energy then all the processors are divided into groups of equal number of

cores while the remaining cores are arranged into one smaller group for one additional energy. The

computation circulates between the different processor groups in such a way that when a group

has completed a T (E) calculation that group receives the next available energy value but waits

until all the groups have completed their calculations. The T (E) values are stored using MPI calls

on one core of each group (sub rank 0) and then stored with one core (global rank 0) in a single

11

array. After all 8,000 T (E) values have been calculated and stored, the processor of rank 0 then

performs the final numerical integration over the stored (E, T (E)) pairs to calculate the total cur-

rent. Benchmarks were all conducted on different high-performance computing resources in Florida

available via the new HPC network Sunshine State Education and Research Computing Alliance

(sserca.org). Details on facilities and available compilers, libraries and software may be found in

the supplementary documentation added.

The plots in Fig. 3 illustrate clearly the nearly optimal scaling behavior of the code for different

levels of MPI parallelism and parallel and distributed Hamiltonian matrix evaluations via PETSc

and MUMPS [47, 48]. The scaling behavior of the code, i.e. the exponential decrease in compute

time with the increase in number of employed compute cores, is demonstrated where the PETSc

and MUMPS libraries are used by each individual core or in a distributed way over two-core groups

per energy step, i.e. the Hamiltonian matrix is inverted for each energy step on a single core or

distributed onto two cores. The scaling behavior is shown for runs on up to 64 or 128 cores,

respectively. The close to perfect scaling behavior for the simulations with two-core versus single-

core matrix inversions can nicely be seen in the logarithmic plots in Fig. 3 for both the use of

the MUMPS and PETSc libraries with visible significant improvement in timing switching from

the MUMPS to PETSc solvers. The system of choice for the benchmarking results are clusters at

Florida State University’s RCC Center as part of Florida’s Sunshine State Education and Research

Computing Alliance (SSERCA) but similar timing for armchair and zigzag nanoring simulations

was observed on the Pegasus cluster at University of Miami’s RCC Center.

Plots

Page 2

1 2 4 8 16 32 64 128

00

8640

17280

25920

34560

43200

Cntor PETSC vs MUMPS Comparison

Armchair

MUMPS Solver – Average MUMPS Solver – Perfect Scaling

PETSC Solver – Average PETSC Solver – Perfect Scaling

Number of Processors

Tim

e (

Se

co

nd

s)

1 2 4 8 16 32 64 128

4

5

6

7

8

9

10

11

Cntor PETSC vs MUMPS Comparison

Armchair

MUMPS Solver – Average MUMPS Solver – Perfect Scaling

PETSC Solver – Average PETSC Solver – Perfect Scaling

Number of Processors

Ln

Tim

e (

Se

co

nd

s)

Plots

Page 8

1 2 4 8 16 32 64 128

00

4320

8640

12960

17280

21600

25920

Cntor PETSC vs MUMPS Comparison

Zigzag

MUMPS Solver – Average MUMPS Solver – Perfect Scaling

PETSC Solver – Average PETSC Solver – Perfect Scaling

Number of Processors

Tim

e (

Se

co

nd

s)

1 2 4 8 16 32 64 128

3

4

5

6

7

8

9

10

11

Cntor PETSC vs MUMPS Comparison

Zigzag

MUMPS Solver – Average MUMPS Solver – Perfect Scaling

PETSC Solver – Average PETSC Solver – Perfect Scaling

Number of Processors

Ln

Tim

e (

Se

co

nd

s)

Fig. 3: Close to perfect scaling of tightbinding code using the PETSc or MUMPS linear solvers onone or two cores for a single matrix inversion at one energy step E. a) Left column of plots: FSUHPC. b) Right column of plots: Univ. of Miami HPC (Pegasus).

The starting tight-binding Hamiltonian is a sparse matrix containing mostly zeros, with the

12

nonzero elements mostly arranged in a banded structure along the diagonal and at the corners of

the matrix. The diagonal and corner matrix elements can be arranged in smaller matrices that

show the same banded structure as the large matrix describing the full Hamiltonian. The additional

corner elements in each case provide ring closure for both the small and large circumference of the

nanoring. (For the simpler carbon nanotube arrangement, those corner elements in the larger matrix

can be set to zero. For a graphene nanoribbon device, in principle the corresponding simplifications

occur for the smaller matrices occur due to removal of the periodic lattice condition also for the

small ring circumference but a modified Hamiltonian describing an effective linearized Dirac-like

electronic behavior on a graphene lattice is more appropriate there.) It is important to note that in

spite of the Hamiltonian matrix H being sparse and banded, the computational problem will scale

with a factor N2 with N as the number of atoms. The PETSc/MUMPS routines utilize two dense

(N,N) matrices to invert H.

3 Library studies on Intel Xeon/Phi and GPUs

3.1 Computational overview

One of the key tasks during this summer 2014 was to work on the code’s scalability and opti-

mize memory usage on different computer architectures using NSF XSEDE, OLCF and regional

resources. The goal was to keep the wall clock time per simulation run of about 8,000 large, sparse

matrix inversions at a manageable level of a few hours per simulation run so a range of different

nanoring chiralities and different low-energy phonon modes maybe studied in a single simulation

run at realistic device dimensions. Physics student Ben Prather at Arizona State University par-

ticipated in the project remotely as part of the 2014 NSF XSEDE Summer Student Engagement

Program under M. Jack as summer research mentor. The effective implementation of optimized

parallel linear solvers such as the PETSc library were studied to optimize code performance in

terms of optimal load balancing, memory usage, speed-up and scalability for system sizes from

roughly 10,000 up to 30,000 atoms. The linking with libraries such as Intel MKL and new tools

like MAGMA [49, 50], developed at the University of Tennessee Knoxville, was investigated with

off-loading of matrix calculations onto Intel Xeon/Phi co-processor cores. Key technical constraint

is that more than 90% of the run time and memory usage of the program consist of these in-

versions. To get an idea of the memory requirements for such a problem size, storing a single

Hamiltonian (or its inverse) as a complex, dense matrix as required by PETSc’s setup e.g. requires

24, 000 × 24, 000 × 16 Bytes = 9.2 GB of memory for a Hamiltonian of 24,000 atoms. An equally

memory-intensive step follows for the actual matrix inversion by direct solution after LU factor-

ization, performed at each energy level. The individual matrix inversions at each energy step can

be treated as independent problems and are parallelized using MPI. Inter-node communication is

to be avoided due to the much longer latencies involved, and matrix calculations are thus to be

performed using all available memory per node.

Currently, there does not exist a library that provides good support for the Intel Xeon/Phi

(MIC) coprocessors in performing all LAPACK functions. Thus, we created a comparative study

13

using five different configurations for linear solvers: a) PETSc with iterative GMRES solver; b)

PETSc with direct MUMPS solver; c) dense solver routines called from the Intel MKL library

linked to PETSc, run entirely on the processor cores; d) off-loading some matrix calculations in

Intel MKL onto the Xeon/Phi co-processor cores; and e) to perform all work on the co-processor

cores via the use of the MAGMA MIC library [49, 50]. For the use of parallelized libraries (PETSc,

Intel MKL, MAGMA MIC), there are two basic strategies for storing a matrix:

• Dense, in which each element receives its own memory, regardless of what value it contains;

• Sparse, in which only a list of nonzero elements and their indices are stored.

Additionally, there are two types of linear algorithms that can be distinguished that either use

a direct solution method with a dense representation of the original matrix and a subsequent LU

factorization step to invert it in place or an iterative solver that uses an initial guess of the solution

via an initial pre-conditioning step for the matrix and then iteratively refines this guess to the

actual solution within some defined error bounds. The timing analysis using MAGMA was limited

to small system sizes of less than 10,000 atoms as the memory available for complete offloading

onto the MIC is limited to 8 GB. Like most matrices, once the Hamiltonian is inverted it contains

very few zeros. Hence, the lower limit on memory use per matrix inversion is that taken by the

stored ’inverse’ Hamiltonian (Green’s function) of 9.2 GB.

There are three classes of algorithms for solving a matrix, once it has been stored:

• Pure dense solvers use a dense representation of the original matrix, then use an LU fac-

torization step and then invert it in place. Since they assume the original matrix could be

anything, they read and multiply all of its zero elements unnecessarily

• Direct solvers try to solve the system A × B = I for B, which is the inverse. In addition to

the initial matrix A, which can be stored sparsely, a direct solver requires storing the solution

B and the identity I as dense matrices , doubling our base memory requirement to 18 GB.

However, it can be faster than a pure dense solver, since it reads only the nonzero values from

A and uses them for the computation.

• Iterative solvers are set up just like direct ones, but use an initial guess for matrix B and then

refine the initial guess until the resulting product A × B is (within some acceptable error)

equal to I. Because the time that is spent for these methods to converge varies based on the

individual algorithm, the pre-conditioner used and the actual sparseness of the initial matrix,

they can be faster or slower than any of the other methods.

Four different configurations were studied:

• PETSc-MUMPS: direct solver, the current method used for the inversions;

• PETSc-GMRES: iterative solver using the same framework;

• MKL: pure dense solver using the LAPACK implementation in the Intel MKL;

14

• MAGMA: pure dense solver using the LAPACK implementation in MAGMA-MIC.

The results for the four studies were the following:

• PETSc-MUMPS: Because this solution requires 18.4 GB of memory, only one inversion prob-

lem fits on each node of Stampede.

• PETSc-GMRES: Memory requirements are the same as for MUMPS, but the solution times

for the GMRES iterative solver are dramatically lower.

• MKL: Memory is halved when using both pure direct methods, to a mere 9.2 GB. Thus

three problems fit on one node instead of two. Scaling works well, but currently off-loading

calculations onto the MIC is available for the LU factorization.

• MAGMA: System sizes with 24,000+ atoms do not fit into the memory for a MIC.

Additionally, a number of other updates and improvements to the code’s user-friendliness were

incorporated: The code was ported to version 3.4 of PETSc, which changed some function calls’

formats and required an explicit call for pre-allocating memory of sparse matrices. Command-line

flags were added to the user may easily set the number of atoms, the number of processes or the

number of threads per single matrix inversion and other parameters. At the end of each simulation

run, the code integrates the electronic transmission rate T (E) (obtained from the Green’s function

after each H inversion) over a range of energy levels E to obtain integrated observables such as

the source-drain current. By reading-in those T (E) values from a file rather than storing them in

memory and by allowing the code to be started at any energy level via a command-line flag, simple

checkpointing is enabled to store intermediate results if the code should crash or stop. The user

may specify the starting energy for a resumed simulation, and the new values will be appended to

the same output file that stores all T (E) values for subsequent integration.

The initial code development was conducted on a SSERCA cluster (Florida State University

RCC). New code builds were compiled for small system sizes to test sparse and dense parallel linear

solvers of the HPC libraries outlined earlier. The Stampede cluster at the Texas Advanced Com-

puting Center then acted as primary resource to perform these scalability and performance tests

for all system sizes using different linear dense and sparse solvers, direct and iterative methods,

multi-threading and off-loading onto Intel Xeon/Phi nodes. Stampede has nodes with 32 GB avail-

able memory shared by 16 processing cores and one Xeon Phi coprocessor per node. An XSEDE

startup allocation on Stampede of roughly 70,000 SUs (SUs=service units) was available to conduct

these tests with a request for an extension by 50,000 SUs recently approved. Stampede possesses

standard 32 GB Intel Xeon/Phi nodes with GPU capability. The NICS/Beacon cluster managed

by the National Institute of Computational Sciences (NICS) at ORNL possesses 48 compute nodes

with 256 GB shared memory, 16 Intel Xeon E5 processors and 4 Intel Xeon/Phi co-processors with

8 GB memory each and is a perfect testbed for large-memory tests and a perfect bridge between

Stampede and Titan with their respective Xeon/Phi and GPU nodes. An additional allocation of

5000 node hours on NICS/Beacon was also available.

15

Fig. 4: Solution times for a single matrix inversion for different nanotorus dimensions and use ofdifferent solvers (here: (10, 10) armchair).

3.2 Benchmarks

The graph for time-stepping of individual matrix inversions for comparisons a) through d) is shown

in Fig. 4. The MUMPS solver for example is very sensitive to exact system sizes and the parallel

storage of the matrix on the cores of each node. For sizes of 12,000 or 24,000 atoms, MUMPS

performed competitively to Intel MKL’s dense solver, which only used half of the memory in the

MUMPS study. MUMPS requires, due to its internal setup, the additional storage of the unit

matrix as dense matrix to later store the dense solution. In the end, PETSc’s iterative solver

GMRES using only one processor core per node to utilize all the node’s shared memory was found

to be faster than any of the other methods listed by roughly a factor of 10, thus prompting our

decision to use this approach for all production runs on Titan.

During a Department of Energy sponsored faculty visit at Oak Ridge National Laboratory

(ORNL) this summer 2014, first production runs to simulate quantum transport including phonon

scattering corrections were started using the Titan resource at ORNL’s Leadership Computing

Facility (OLCF) courtesy of summer host B. Sumpter after an initial test phase in early June on the

OIC/Phase 5 cluster at ORNL’s Center for Nanophase Materials Sciences (CNMS). In the initial

test phase on OIC/Phase 5, simulation runs were performed for small system sizes (10,000 atoms)

using about 2 to 8 nodes with 16-core nodes before switching to OLCF/Titan for first successful

production runs for nanorings from 3,600 to 24,000 atoms and a range of different armchair and

zigzag chiralities. Titan’s general scheduling policy promoting computational jobs with very large

node count but compute times restricted to one-hour runs proves optimal for our production runs.

A sketch of typical node counts n per job on Titan for different system sizes N and estimated SUs

16

spent is listed below:

N = 3600− 12000; armchair: n = 64; zigzag: n = 128.N = 12000− 24000; armchair: n = 128− 256; zigzag: n = 256− 512.

Example: (6, 0) zigzag: N = 24000; n = 1024.Max. run time: 1 hour; SUs per run: 2048− 8192 SUs.

3.3 Physics production runs

Initial simulation runs for a range of armchair and zigzag chiralities and ring sizes nicely reproduce

van-Hove singularities in form of spikes in the density-of-states D(E) and transmissivity T (E). Van-

Hove singularities correspond to resonances in transmission of the interfering electronic pathways

on the torus surface with carrier injection and extraction from the attached metallic leads. To be

exact, 1D conductors will show spikes in the density-of-states D(E) that diverge like 1√E

at band

edges due to electronic confinement along the small torus circumference [3]. In Fig. 5a for example

the density-of-states are depicted with significant enhancements visible at resonance energies acting

as available transmission channels. Armchair chiralities in Fig. 5a are (3,3), (5,5) and (10, 10) for

N = 3, 600 carbon atoms. Rings along the minor circumference include 12, 20, and 40 atoms

and rings along the major circumference 300, 180, and 90 atom, respectively. Resonance energies

depicted in Fig. 5a clearly scale with 1/R with R being the major ring radius. These signatures

can clearly be reproduced with the new parallel code for all ring sizes and all armchair tori and

those zigzag tori that are metallic, i.e. conducting. The general prescription that carbon nanotubes

with chiralities (m,n), that satisfy mod[(m − n)/3] = 0, are metallic carries over to the toroidal

armchair (n, n) and zigzag (n, 0) cases. Simulations of zigzag rings with mod[(m − n)/3] = 1 or

2 do not show any transport but semi-conducting behavior as expected. The density-of-states

D(E) vanishes at the Fermi energy [3]. Additionally, width-dependent effects such as a splitting of

van-Hove singularities can be observed where transport resonance for metallic tori along the inner

or outer radius of the torus surface creates a splitting of a van-Hove singularity into two distinct

peaks leading to a drop-off in transmissivity T (E) by several orders of magnitude between peaks.

The effect is more pronounced with increasing torus width relative to the large torus diameter.

The splitting of resonances is highlighted in Fig. 5b for example for a (10, 10) armchair torus of

3,600 atoms: A van-Hove singularity at E = 0.187 eV transforms into two symmetric peaks with

a roughly 3 meV splitting, a 2%-effect for a ring width of 2a = 13.6 A and a torus diameter of

2R = 35 A.

4 Results

A summary of all benchmarking results for different linear solvers and libraries or different com-

puting models (only on processor cores, off-loading on Intel Xeon/Phi co-processors or GPUs, or

symmetric) completed with XSEDE summer student Ben Prather this summer and fall 2014 is

shown in Fig. 6 below. Simulations have been conducted both on NSF XSEDE (TACC/Stampede)

and DoE OLCF (Titan, NICS/Beacon, OIC/Phase5) resources for different nanoring dimensions,

17

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●●

●

●

●

●

●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

−0.2 −0.1 0.0 0.1 0.2

1e−

031e

−01

1e+

011e

+03

Energy E (eV)

Den

sity

D(E

)Density of states by energy at 3600 atoms

● ● ●

Chiralities

33 55 1010

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

−0.200 −0.195 −0.190 −0.185 −0.180

1e−

021e

+00

1e+

021e

+04

Energy E (eV)

Den

sity

D(E

)

Density of states by energy at (10,10), 3600 atoms

Fig. 5: a) Density-of-states D(E) with van-Hove singularities for (3,3), (5,5) and (10, 10) arm-chair nanorings (N=3,600). b) Splitting of a van-Hove singularity in (10, 10) armchair nanorings(N=3,600).

5000 10000 15000 20000 250000

50

100

150

200

250

300

350

400

450

Solution Time vs Problem Size for Various Solvers

GMRES

MUMPS-thd

MUMPS

MKL

MKL-MIC

MAGMA

MAG-MIC

Size of system (atoms)

Tim

e to

sol

utio

n of

one

ene

rgy

step

(s)

Fig. 6: Comparison of compute times per matrix inversion (single energy step) for different linearsolvers, libraries and computing models.

chiralities, device setups and phonon modes. All library benchmarks have been performed and

recorded this summer on Intel Xeon/Phi and GPU architectures for symmetric, heterogeneous and

offloading programming models to study optimal code performance for system sizes of 10,000 to

30,000 atoms. For our tightbinding sparse-matrix description the traditional GMRES iterative lin-

ear solver using one core and all the memory per node still proved to be the most effective approach

by far.

5 Conclusions

The new parallel code QRing correctly predicts quantum transport features in carbon nanoring

devices including van Hove singularities and width effects. It is scalable to armchair/zigzag systems

withN = 30000 atoms and versatile to model nanoring, nanotube, and graphene nanoribbon setups.

18

The general use of HPC linear solver libraries like PETSc allows for a later extension to higher-order

approximations beyond tightbinding. Following physics program is scheduled:

• To model low-energy phonon corrections with effective linear electron-phonon coupling pro-

vided in a collaboration with Georgia Institute of Technology (M.J. Leamy) via a continuum

method using a finite-difference time domain approach [25, 26];

• To place the nanodevice on top of metallic leads for o(100×) larger current amplitudes;

• To study the stability of modal symmetries in coherent transport under electron-phonon

coupling to low-energy phonon modes [51];

• To include magnetic-field dependence for magneto-transport studies;

• To extend to other quasi-particle corrections such as exciton-plasmon coupling etc. [22, 23,

46, 24].

On the technical side, it is planned to use available open-source and commercial parallel profiling

tools such as TAU to study parallel thread calls on Intel Xeon/Phi cards and how manipulating

these impacts the code performance, to test other iterative solvers with added pre-conditioning,

and to investigate further code improvements with PETSc version 3.5.0 linked to the new sparse

solver PARDISO [52, 53]. These studies may be continued with MAGMA on GPUs using CUDA

compilers and tools such as CUBlas.

Acknowledgments

Funding for this project was provided in parts by the 2014 XSEDE Summer Student Program

(BP) and the Shodor/Blue Waters Undergraduate Petascale Education Program (AB, LD). MJ

would like to acknowledge the support by the 2014 ORISE Visiting Faculty Program at Oak Ridge

National Laboratory (ORNL) funded by the Department of Energy. MJ would further like to

thank the ORNL Center for Nanophase Materials Sciences and the Sunshine State Education

and Research Computing Alliance (SSERCA) for available computational resources and helpful

discussions. The team would like to thank in particular collaborators M.J. Leamy at Georgia

Institute of Technology, J. Jakowski at the University of Tennessee Knoxville, J. Oelgoetz at Austin

Peay State University, and S. Khondaker at the University of Central Florida, and J. Wilgenbusch at

the Minnesota Supercomputing Institute at the University of Minnesota for helpful discussions and

support. This work used the Extreme Science and Engineering Discovery Environment (XSEDE),

which is supported by National Science Foundation grant number ACI-1053575. This research used

resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User

Facility supported under Contract DE-AC05-00OR22725.

19

References

[1] R.R. Hartmann, J. Kono, and M.E. Portnoi. Terahertz science and technology of carbon

nanomaterials. Nanotechnology 25, page 322001, 2014.

[2] F. Bonacorso. Graphene photonics and optoelectronics. Nat. Photonics, 6(749), 2010.

[3] J.C. Charlier, X. Blase, and S. Roche. Electronic and transport properties of carbon nanotubes.

Rev. Mod. Phys., 79:677, 2009.

[4] A.H. Castro Neto, F. Guinea, N.M.R. Peres, K.S. Novoselov, and A.K. Geim. The electronic

properties of graphene. Rev. Mod. Phys., 81:109, 2009.

[5] H. Suzuura and T. Ando. Phonons and electron-phonon scattering in carbon nanotubes. Phys.

Rev., B65:235412, 2002.

[6] J.-Y. Park, S. Rosenblatt, Y. Yaish, V. Sazonova, H. Ustunel, S. Braig, T. A. Arias, Piet W.

Brouwer, and P.L. McEuen. Electron-Phonon Scattering in Metallic Single-Walled Carbon

Nanotubes. Nano Lett. 4, 4(517), 2004.

[7] S. Iijima. Helical microtubules of graphitic carbon. Nature (London), 354:56, 1991.

[8] W. Choi, S. Hong, J.T. Abrahamson, J.-H. Han, C. Song, N. Nair, S. Baik, and M. S. Strano.

Chemically driven carbon-nanotube-guided thermopower waves. Nature Mater., 9:423, 2010.

[9] S. Motavas, B. Omrane, and C. Papadopoulos. Large area patterning of carbon nanotube ring

arrays. Langmuir, 25(8):4655, 2009.

[10] G.N. Afanasiev and V.M. Dubovik. Some remarkable charge-current configurations. Phys.

Part. Nucl., 29:366, 1998.

[11] G.N. Afanasiev. Simplest sources of electromagnetic fields as a tool for testing the reciprocity-

like theorems. J. Phys., D34:539, 2001.

[12] A.D. Boardman, K. Marinov, N. Zheludev, and V.A. Fedotov. Dispersion properties of non-

radiating configurations: Finite-difference time-domain modeling. Phys. Rev., E72:036603,

2005.

[13] N. Papasimakis, V.A. Fedotov, K. Marinov, and N.I. Zheludev. Gyrotropy of a metamolecule:

wire on a torus. Phys. Rev. Lett., 103:093901, 2009.

[14] T. Kaelberer, V.A. Fedotov, N. Papasimakis, D. P. Tsai, and N.I. Zheludev. Toroidal dipolar

response in a metamaterial. Science, 330:1510, 2010.

[15] L. Liu, G.Y. Guo, C.S. Jayanthi, and S. Y. Wu. Colossal paramagnetic moments in metallic

carbon nanotori. Phys. Rev. Lett., 88:217206, 2002.

20

[16] K. Sasaki and Y. Kawazoe. Characteristic behavior of toroidal carbon nanotubes: Kinematics

of persistent currents. Prog. Theor. Phys, 112(3):369, 2004.

[17] M. Encinosa and M. Jack. Elliptical tori in a constant magnetic field. Physica Scripta, 73:439,

2006.

[18] M. Encinosa and M. Jack. Dipole and solenoidal magnetic moments of electronic surface

currents on toroidal nanostructures. Journal of Computer-Aided Materials Design, 14(1):65,

2007.

[19] J. Williamson and M. Encinosa. Quantum toroidal moments of an elliptic toroidal helix in a

constant magnetic field. quant-ph/1108.5097.

[20] M. Encinosa and J. Williamson. Toroidal moments of schrodinger eigenstates.

quant-ph/1106.4248v1.

[21] F. H. L. Koppens et al. Graphene plasmonics: A platform for strong light-matter inter-actions.

Nano Lett. 11, 11(3370), 2011.

[22] G. Bryant. Hubbard model for intermediate dimensional excitons. Phys. Rev., B49(23):16129,

1994.

[23] V.N. Popov and L. Henrard. Comparative study of the optical properties of single-walled

carbon nanotubes within orthogonal and nonorthogonal tight-binding models. Phys. Rev.,

B70:115407, 2004.

[24] I.V. Bondarev, L.M. Woods, and K. Tatur. Strong exciton-plasmon coupling in semiconducting

carbon nanotubes. Phys. Rev., B80:085407, 2009.

[25] M.J. Leamy and A. DiCarlo. Phonon spectra prediction in carbon nanotubes using a manifold-

based continuum finite element approach. Computer Methods in Applied Mechanics and En-

gineering, 198(17-20):1572, 2009.

[26] M.J. Leamy. Calculation of phonon dispersion in carbon nanotubes using a continuum-

atomistic finite element approach. AIP Advances, 1:041702, 2011.

[27] S. Balay, J. Brown, K. Buschelman, V. Eijkhout, W. Gropp, D. Kaushik, M. Knepley, L. Curf-

man McInnes, B. Smith, and H. Zhang. Petsc users manual. PETSc 4.3, ANL-95/11, October

15, 2013. http://www.mcs.anl.gov/petsc.

[28] G. Klimeck, S.S. Ahmed, H. Bae, N. Kharche, and R. Rahman. Atomistic simulation of

realistically sized nanodevices using NEMO 3-d: Part i - models and benchmarks. Special Issue

on Nanoelectronic Device Modeling in IEEE Transactions on Electron Devices, 54(9):2079,

2007.

21

[29] G. Klimeck, S.S. Ahmed, N. Kharche, M. Korkusinki, and M. Usman. Atomistic simulation

of realistically sized nanodevices using NEMO 3-d: Part II - applications, special issue on

nanoelectronic device modeling. IEEE Transactions on Electron Devices, 5(9):2090, 2007.

[30] B.P. Haley, S. Lee, M. Luisier, H. Ryu, F. Saied, S. Clark, H. Bae, G. Klimeck, and F. Saied.

Advancing nanoelectronic device modeling through peta-scale computing and deployment on

nanoHUB. J. Phys. : Conf. Ser, 180:012075, 2009.

[31] M. Luisier and G. Klimeck. Atomistic full-band simulations of silicon nanowire transistors:

Effects of electron-phonon scattering. Phys. Rev., B94(22):155430, 2009.

[32] M. Luisier and G. Klimeck. Performance analysis of statistical samples of graphene nanoribbon

tunneling transistors with line edge roughness. Appl. Phys. Lett., 94(22):223505, 2009.

[33] S. Kim, M. Luisier, T.B. Boykin, and G. Klimeck. Computational study of heterojunction

graphene nanoribbon tunneling transistors with p-d orbital tight-binding method. Appl. Phys.

Lett., 104(24):224113, 2014.

[34] B. Aradi, B. Hourahine, and Th. Frauenheim. DFTB+, a sparse matrix-based implementation

of the DFTB method. J. Phys. Chem., A111(26):5678, 2007.

[35] G. Zheng, M. Lundberg, J. Jakowski, T. Vreven, M.J. Frisch, and K. Morokuma. Implemen-

tation and benchmark tests of the DFTB method and its application in the ONIOM method.

Int. J. Quant. Chem., 109(9):1842, 2009.

[36] S. Garashchuk, J. Jakowski, L. Wang, and B.G. Sumpter. Quantum trajectory-electronic

structure approach for exploring nuclear effects in the dynamics of nanomaterials. J. Chem.

Theory Comput., 9:5221, 2013.

[37] L. Wang, J. Jakowski, and S. Garashchuk. Adsorption of a hydrogen atom on a graphene

flake examined with quantum trajectory/electronic structure dynamics. J. Phys. Chem.,

C118:16175, 2014.

[38] V. Perebeinos, J. Tersoff, and P. Avouris. Effect of exciton-phonon coupling in the calculated

optical absorption of carbon nanotubes. Phys. Rev. Lett., 94:027402, 2005.

[39] L.M. Woods and G.D. Mahan. Electron-phonon effects in graphene and armchair (10, 10)

single-wall carbon nanotubes. Phys. Rev., B61(16):10651, 2000.

[40] L.V. Keldysh. Diagram technique for nonequilibrium processes. Sov. Phys. JETP, 20(4):1018,

1965.

[41] A.P. Jauho, N.S. Wingreen, and Y. Meir. Time-dependent transport in interacting and non-

interacting resonant-tunneling systems. Phys. Rev., B50:5528, 1994.

[42] S. Datta. Electronic Transport in Mesoscopic Systems. Cambridge University Press (NY),

1995.

22

[43] M.P. Anantram and T.R. Govindan. Conductance of carbon nanotubes with disorder: A

numerical study. Phys. Rev., B58(8):4882, 1998.

[44] M. Encinosa and M. Jack. Quantum electron transport in toroidal carbon nanotubes with

metallic leads. Journal of Molecular Simulations, 34(1):9, 2008.

[45] M. Encinosa and M. Jack. Excitation of surface dipole and solenoidal modes on toroidal

structures. physics/0604214.

[46] I.V. Bondarev, K. Tatur, and L.M. Woods. Surface exciton-plasmons and optical response of

small-diameter carbon nanotubes. Opt. and Spectr, 108:376, 2010.

[47] P.R. Amestoy, I.S. Duff, J. Koster, and J.-Y. L’Excellent. A fully asynchronous multifrontal

solver using distributed dynamic scheduling. SIAM Journal of Matrix Analysis and Applica-

tions, 23(1):15, 2001.

[48] P.R. Amestoy, A. Guermouche, J.-Y. L’Excellent, and S. Pralet. Hybrid scheduling for the

parallel solution of linear systems. Parallel Computing, 32(2):136, 2006.

[49] E. Agullo, J. Demmel, J. Dongarra, B. Hadri, J. Kurzak, J. Langou, H. Ltaief, P. Luszczek, and

S. Tomov. Numerical linear algebra on emerging architectures: The PLASMA and MAGMA

projects. J. Phys. : Conf. Ser., 180(1):012037, 2009.

[50] J. Dongarra, M. Gates, A. Haidar, Y. Jia, K. Kabir, P. Luszczek, and S. Tomov. Portable

HPC Programming on Intel Many-Integrated-Core Hardware with MAGMA Port to Xeon Phi.

PPAM 2013, Warsaw, Poland, Sep 2013.

[51] K.R. Dienes and B. Thomas. Isospectral But Physically Distinct: Modular Symmetries and

their Implications for Carbon Nanotori. Phys. Rev., B84:085444, 2011.

[52] A. Kuzmin, M. Luisier, and O. Schenk. Fast Methods for Computing Selected Elements of

the Green’s Function in Massively Parallel Nanoelectronic Device Simulations. Euro-Par 2013,

LNCS 8097, F. Wolf and B. Mohr and D. an Ney (Eds.), Springer-Verlag Berlin Heidelberg,

2013.

[53] C. Petra, O. Schenk, M. Lubin, and K. Gartner. An augmented incomplete factorization ap-

proach for computing the Schur complement in stochastic optimization. SIAM J. Sci. Comput,

36(2):C139, 2014.

23