ORNL is managed by UT-Battelle for the US Department of Energy

Titan Cross Compile

Adam Simpson

OLCF User Support

2 Cross Compiling

Titan node structure - connectivity

ExternalLogin

ServiceNodes

Internet

SSH

aprunqsub

ComputeNodes

3 Cross Compiling

Titan node structure - processor

• External Login– 4 x 8 core AMD Magny-Cours

• Service Nodes– 6 core AMD Istanbul

• Compute Nodes– 16 core AMD Interlagos

4 Cross Compiling

Cray Compile Wrapper

• CC, cc, ftn

• Controlled by xt-asyncpe module

• Sets compiler based on PrgEnv-* module– Same command for PGI, GNU, Cray, Intel

• Sets optimizations and target processor/arch.

• Adds libraries based on loaded Cray modules– Networking/Communication libraries(MPI, GNI, PMI, etc.)– Science Libraries(CUDA, LibSci, NetCDF, ...)– ALPS libraries

5 Cross Compiling

Compile for login/service nodes

• Don’t run wrapper compiled code on non compute nodes– Illegal instruction

• Interlagos has AVX/SSE4 instructions

– Missing shared libraries• Error while loading shared libraries: libalpslli.so.0

6 Cross Compiling

Compile for login/service nodes

• module load craype-target-local_host– Compile for external login / service node*– Doesn’t add Cray specific libraries– Doesn’t work as advertised

• module load craype-target-native– Target node common instruction set– Doesn’t add any module related libraries

• No MPI, LibSci, CUDA, etc.

• Bypass wrappers– pgcc, pgCC, pgfortran : gcc, g++, gfortran– craycc, crayc++, crayftn : icc, icpc, ifort

7 Cross Compiling

Compile for login/service nodes

• If Cray libraries required on external logins– module switch craype-interlagos craype-mc8

• If Cray libraries required on service nodes– module switch craype-interlagos craype-istanbul

8 Cross Compiling

CLE 5.2 Changes

• xt-asyncpe: deprecated– renamed craype

• craype-target-*: deprecated– craype-interlagos– craype-istanbul– craype-mc8– craype-network-gemini– craype-network-none