Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Linux Clusters InstituteLmod A Modern
Environment Module SystemRobert McLay Manager Software Tools TACC
1
Outlinebull What are Environment Modulesbull Kinds of Module Tools Tmod Cmod Lmod hellipbull What is Lmodbull Modulefile Architecturebull Advanced Topicsbull Where to go for help
May 2017 2
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 3
What are Modulesbull Modules are the way sites provide optional software MPI Boost bull Modules add to PATH and set other env vars for each packagebull Modules can be unloadedbull Modulefiles are in one file not one for each shell (eg Compiler init scripts)
May 2017 4
What is Lmodbull A modern replacement for a tried and true conceptbull The guiding principal ``Make life easier wo getting in the wayrsquobull Reads both TCL and Lua modulefiles
May 2017 5
Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the
same compilerbull Similarly MPI Applications must build and link with same MPICompiler
pairing when using pre-built MPI libraries
May 2017 6
Example of Lmod Environment Modules (I)
$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------
petsc31 (default) petsc31-debug pmetis40 tau2203
------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143
boost1460 mpich2132 openmpi151
boost1461 (default) mpich214 (default) openmpi153 (default)
-------------------------- optappsmodulefilesCore ----------------------
StdEnv intel111 papi414
adminadmin-10 intel120 (default) scite228
ddtddt lmodlmod tex2010
dmallocdmalloc locallocal (default) unixunix (default)
fdepend12 mklmkl visitvisit
gcc44 noweb211b
gcc45 (default)
May 2017 7
Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31
$ module unload gccInactive Modules1) mpich2 2) petsc
$ module load intelActivating Modules1) mpich2 2) petsc
$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc
May 2017 8
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Outlinebull What are Environment Modulesbull Kinds of Module Tools Tmod Cmod Lmod hellipbull What is Lmodbull Modulefile Architecturebull Advanced Topicsbull Where to go for help
May 2017 2
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 3
What are Modulesbull Modules are the way sites provide optional software MPI Boost bull Modules add to PATH and set other env vars for each packagebull Modules can be unloadedbull Modulefiles are in one file not one for each shell (eg Compiler init scripts)
May 2017 4
What is Lmodbull A modern replacement for a tried and true conceptbull The guiding principal ``Make life easier wo getting in the wayrsquobull Reads both TCL and Lua modulefiles
May 2017 5
Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the
same compilerbull Similarly MPI Applications must build and link with same MPICompiler
pairing when using pre-built MPI libraries
May 2017 6
Example of Lmod Environment Modules (I)
$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------
petsc31 (default) petsc31-debug pmetis40 tau2203
------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143
boost1460 mpich2132 openmpi151
boost1461 (default) mpich214 (default) openmpi153 (default)
-------------------------- optappsmodulefilesCore ----------------------
StdEnv intel111 papi414
adminadmin-10 intel120 (default) scite228
ddtddt lmodlmod tex2010
dmallocdmalloc locallocal (default) unixunix (default)
fdepend12 mklmkl visitvisit
gcc44 noweb211b
gcc45 (default)
May 2017 7
Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31
$ module unload gccInactive Modules1) mpich2 2) petsc
$ module load intelActivating Modules1) mpich2 2) petsc
$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc
May 2017 8
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 3
What are Modulesbull Modules are the way sites provide optional software MPI Boost bull Modules add to PATH and set other env vars for each packagebull Modules can be unloadedbull Modulefiles are in one file not one for each shell (eg Compiler init scripts)
May 2017 4
What is Lmodbull A modern replacement for a tried and true conceptbull The guiding principal ``Make life easier wo getting in the wayrsquobull Reads both TCL and Lua modulefiles
May 2017 5
Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the
same compilerbull Similarly MPI Applications must build and link with same MPICompiler
pairing when using pre-built MPI libraries
May 2017 6
Example of Lmod Environment Modules (I)
$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------
petsc31 (default) petsc31-debug pmetis40 tau2203
------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143
boost1460 mpich2132 openmpi151
boost1461 (default) mpich214 (default) openmpi153 (default)
-------------------------- optappsmodulefilesCore ----------------------
StdEnv intel111 papi414
adminadmin-10 intel120 (default) scite228
ddtddt lmodlmod tex2010
dmallocdmalloc locallocal (default) unixunix (default)
fdepend12 mklmkl visitvisit
gcc44 noweb211b
gcc45 (default)
May 2017 7
Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31
$ module unload gccInactive Modules1) mpich2 2) petsc
$ module load intelActivating Modules1) mpich2 2) petsc
$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc
May 2017 8
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
What are Modulesbull Modules are the way sites provide optional software MPI Boost bull Modules add to PATH and set other env vars for each packagebull Modules can be unloadedbull Modulefiles are in one file not one for each shell (eg Compiler init scripts)
May 2017 4
What is Lmodbull A modern replacement for a tried and true conceptbull The guiding principal ``Make life easier wo getting in the wayrsquobull Reads both TCL and Lua modulefiles
May 2017 5
Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the
same compilerbull Similarly MPI Applications must build and link with same MPICompiler
pairing when using pre-built MPI libraries
May 2017 6
Example of Lmod Environment Modules (I)
$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------
petsc31 (default) petsc31-debug pmetis40 tau2203
------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143
boost1460 mpich2132 openmpi151
boost1461 (default) mpich214 (default) openmpi153 (default)
-------------------------- optappsmodulefilesCore ----------------------
StdEnv intel111 papi414
adminadmin-10 intel120 (default) scite228
ddtddt lmodlmod tex2010
dmallocdmalloc locallocal (default) unixunix (default)
fdepend12 mklmkl visitvisit
gcc44 noweb211b
gcc45 (default)
May 2017 7
Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31
$ module unload gccInactive Modules1) mpich2 2) petsc
$ module load intelActivating Modules1) mpich2 2) petsc
$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc
May 2017 8
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
What is Lmodbull A modern replacement for a tried and true conceptbull The guiding principal ``Make life easier wo getting in the wayrsquobull Reads both TCL and Lua modulefiles
May 2017 5
Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the
same compilerbull Similarly MPI Applications must build and link with same MPICompiler
pairing when using pre-built MPI libraries
May 2017 6
Example of Lmod Environment Modules (I)
$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------
petsc31 (default) petsc31-debug pmetis40 tau2203
------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143
boost1460 mpich2132 openmpi151
boost1461 (default) mpich214 (default) openmpi153 (default)
-------------------------- optappsmodulefilesCore ----------------------
StdEnv intel111 papi414
adminadmin-10 intel120 (default) scite228
ddtddt lmodlmod tex2010
dmallocdmalloc locallocal (default) unixunix (default)
fdepend12 mklmkl visitvisit
gcc44 noweb211b
gcc45 (default)
May 2017 7
Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31
$ module unload gccInactive Modules1) mpich2 2) petsc
$ module load intelActivating Modules1) mpich2 2) petsc
$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc
May 2017 8
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Fundamental Issuesbull Software Packages are created and updated all the timebull Some Users need new versions for new features and bug fixesbull Other Users need older versions for stability and continuitybull User programs using pre-built C++ amp Fortran libraries must link with the
same compilerbull Similarly MPI Applications must build and link with same MPICompiler
pairing when using pre-built MPI libraries
May 2017 6
Example of Lmod Environment Modules (I)
$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------
petsc31 (default) petsc31-debug pmetis40 tau2203
------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143
boost1460 mpich2132 openmpi151
boost1461 (default) mpich214 (default) openmpi153 (default)
-------------------------- optappsmodulefilesCore ----------------------
StdEnv intel111 papi414
adminadmin-10 intel120 (default) scite228
ddtddt lmodlmod tex2010
dmallocdmalloc locallocal (default) unixunix (default)
fdepend12 mklmkl visitvisit
gcc44 noweb211b
gcc45 (default)
May 2017 7
Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31
$ module unload gccInactive Modules1) mpich2 2) petsc
$ module load intelActivating Modules1) mpich2 2) petsc
$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc
May 2017 8
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Example of Lmod Environment Modules (I)
$ module avail------------------ optappsmodulefilesMPIintel120mpich214 ---------
petsc31 (default) petsc31-debug pmetis40 tau2203
------------------- optappsmodulefilesCompilerintel120 --------------boost1450 gotoblas2113 openmpi143
boost1460 mpich2132 openmpi151
boost1461 (default) mpich214 (default) openmpi153 (default)
-------------------------- optappsmodulefilesCore ----------------------
StdEnv intel111 papi414
adminadmin-10 intel120 (default) scite228
ddtddt lmodlmod tex2010
dmallocdmalloc locallocal (default) unixunix (default)
fdepend12 mklmkl visitvisit
gcc44 noweb211b
gcc45 (default)
May 2017 7
Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31
$ module unload gccInactive Modules1) mpich2 2) petsc
$ module load intelActivating Modules1) mpich2 2) petsc
$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc
May 2017 8
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Example of Lmod Environment Modules (II)$ module listCurrently Loaded Modules1) StdEnv 2) gcc45 3) mpich214 4) petsc31
$ module unload gccInactive Modules1) mpich2 2) petsc
$ module load intelActivating Modules1) mpich2 2) petsc
$ module load gccDue to MODULEPATH changes the follow modules have been reloaded1) mpich2 2) petsc
May 2017 8
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Why You Might Want To Use Lmod
bull Same module commands as in Tmodbull Active Development Frequent Releases Bug fixesbull Vibrant Communitybull Used all over the worldbull Enjoy many capabilities wo changing a single module filebull Many more advantages when youre ready
May 2017 9
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Lmod Features
bull Reads for TCL and Lua modulefilesbull One name rulebull Support a Software Hierarchybull Fast module avail via optional spider cachebull Properties (gpu mic)bull Semantic Versioning 56 is older than 510bull family(``compiler) family(``MPI) supportbull Optional Tracking What modules are usedbull Many other features ml collections hooks nag
May 2017 10
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Tmod vs Lmod
bull Tmod (TCLC) is in maintenance mode Lmod activebull Some work on Tmod (TCL) recentlybull Lmod has many more featuresbull Tmod module load gcc53 gcc60 loads both modulesbull Lmod has the ``One Name Rulersquobull Lmod is close to Tmod but not the same
May 2017 11
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Safety Features of Lmod (I)bull Users can only load one version of a packagebull module load xyz21 -gt load xyz version 21bull module load xyz22 -gt unload 21 loads 22bull This can not be overridden
May 2017 12
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Safety Features of Lmod (II)bull Lmod added a new command family(rdquonamerdquo)bull All of our compiler modules have family(ldquocompilerrdquo)bull All of our MPI modules have family(ldquoMPIrdquo)bull Users can only load one compiler or MPI stack at a timebull Power users can override this restriction at their own peril
May 2017 13
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Module Architecture Designbull Lua or TCL modulefilesbull Optional software shared or localbull Flat or Hierarchical Module Layoutbull Naming conventions (NV CNV NVV)bull A standard set of modulesbull Keep software for life of cluster or notbull Problems with Bash
May 2017 14
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Shared Disk versus Local Installbull Local install Fast smallbull Shared Disk Big slowerbull TACC local install (mostly)bull Shared Disk Keep software for life of system
May 2017 15
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Modulefile Layout Choicesbull Flat Naming Schemebull Hierarchical Naming Scheme
May 2017 16
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Flat Naming Scheme PETScbull PETSc is a parallel iterative solver package
bull PETSc41-mvapich2-21-gcc-63bull PETSc41-mvapich2-21-intel-170bull PETSc41-openmpi-18-gcc-63bull PETSc41-openmpi-18-intel-170
May 2017 17
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Problems w Flat naming scheme
bull User have to loadbull module load intel170bull module load mvapich221-intel-170bull module load PETSc41-mvapich2-21-intel-170
bull Changing compilers means unloading all three modulesbull Reloading new compiler MPI PETSc modulesbull Not loading correct modules -gt Mysterious Failuresbull Onus on package compatibility on usersbull Or extremely complicated modulefiles
May 2017 18
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Extremely complicated modulefilesbull Protect users via conflicts andor prereqsbull The problem is that they are fragilebull What happens with a new compiler or MPI stack
May 2017 19
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Hierarchical Naming Schemesbull Store modules under one tree optappsmodulefilesbull One strategy is to use sub-directories
bull Core Regular packages apps compilers gitbull Compiler Packages that depend on compiler boost MPIbull MPI Packages that depend on MPICompiler PETSc FFTW3
May 2017 20
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
MODULEPATH
bull MODULEPATH is a colon separated list of directories containing directories and module files
bull No modulefiles loaded -gtusers can only load core modulesbull Loading a compiler module adds to MODULEPATH
bull Users can load compiler dependent modulesbull This includes MPI implementations modules
bull Loading an MPI module adds to MODULEPATHbull Users can load MPI libraries that match the MPIcompiler pairing
bull See httpslmodreadthedocsioenlatest080_hierarchyhtml for details
May 2017 21
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Modulefile contentsbull Be consistent Find a convention and stick with itbull We define consistent variables in each modulebull ltSITE_NAMEgt_ltPKG_NAMEgt_LIBINCBIN
bull TACC_HDF5_BINbull TACC_HDF5_INC
May 2017 22
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Providing a standard set of modulesbull Define a compiler and mpi stackbull optappsmodulefilesCoreStdEnvlua
bull load(gccmvapich2)
May 2017 23
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
A Standard Set of Modules (II)bull In etcprofiledz99_StdEnvsh
if [ -z $__Init_Default_Modules ] thenexport __Init_Default_Modules=1export LMOD_SYSTEM_DEFAULT_MODULES=StdEnvrdquomodule --initial_load --no_redirect restore
elsemodule refresh
fibull Why is the module refresh there
May 2017 24
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
User Collections with SaveRestore
bull Users can setup their own initially loaded modulesbull Users simply load unload andor swap until happybull module save saves state into ldquodefaultrdquobull Shell startup scripts do ldquomodule restorerdquo which load userrsquos default if it
existsbull Users can create other collections
bull $ module save name to save it bull $ module restore name to restore it
bull Note that a collection does a module purge before restoring collection
May 2017 25
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Module reset restorebull module reset -gt module purge module load $LMOD_SYSTEM_DEFAULT_MODULESbull module restore -gt module purge load default collection or module reset
May 2017 26
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Module Naming Conventionsbull NV NameVersion (eg bowtie23)bull CNV CategoryNameVersion (eg biobowtie23)bull NVV NameVersionVersion (eg bowtie6423)
May 2017 27
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Module Naming Conventions (II)
bull Try to stick with NV if possiblebull Its less typingbull CNV might be helpful to novice usersbull But your obvious categories may not be obvious to your usersbull Avoid NVV unless your users are expertsbull Or if you really need 6432 bit libraries
May 2017 28
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Bash issues
bull Bash Startup is typically ``broken for non-login interactive shellsbull Redhat Centos MacOS typically DO NOT source etcbashrc on interactive
shellsbull MPI jobs start an interactive shell
May 2017 29
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Bash Issues (II)
bull Want module command to work in all shellsbull Want stacksize unlimited for MPI jobsbull We patched bash to force it to source etctaccbashrc
May 2017 30
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Bash Repair Choices
bull Switch users to Z shell bull patch bash (see Lmod docs) bull Expect all users to source etcbashrc in ~bashrcbull Expect all users to start jobs with binbash -lbull I donrsquot trust all users to do the right thing trade
May 2017 31
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Keeping Software for Life of Cluster or Not
bull It is possible with a shared disk approachbull You might want to hide older modules
May 2017 32
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Hidden modules
bull Sites have always hide modules by adding a leading dotbull For example gcc63bull Lmod 7 also allows for hidden module via MODULERCbull system MODULERC or ~modulerc
May 2017 33
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Using System MODULERC to hide modules
bull In $MODULERCFILE or applmodetcrcModulehide-version foo32
May 2017 34
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Tracking Module Usage
bull Lmod makes it easy to track module usagebull Lmod can be setup to send a tagged message to syslogbull Rsyslog can send tags to a separate filebull See lmodcontribtracking_module_usage for details
May 2017 35
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Usage counts
$ analyzeLmodDB --sqlPattern fftw --start 2015-01-01rsquo --end2015-02-01 countsModule path Distinct Users----------- --------------appsintel13mpich_3_2mfilesfftw3332 151 appsintel13mpich_3_2mfilesfftw2215 62 appsintel13impi_4_1mfilesfftw3332 45 appsintel13impi_4_1mfilesfftw2215 19
May 2017 36
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Distinct Users
$ analyzeLmodDB --sqlPattern settarg usernameModule path User Name----------- ---------appsmfilessettarg58 user1appsmfilessettarg58 user2appsmfilessettarg58 user3appsmfilessettarg581 mclayappsmfilessettarg591 user5
May 2017 37
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Why does Lmod work at all
bull The Environment is inherited from the parent processbull Changes in the childs environment DOES NOT affect the parentsbull So how could Lmod work at all
May 2017 38
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
The trick is
bull The lmod program generates textbull The module command evals that textbull module () eval $( $LMOD_CMD bash ldquo$rdquo)
May 2017 39
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Why is this important
bull Its a useful trick to knowbull Debugging Modulefilesbull $LMOD_CMD bash load module 2gt devnull gt stdouttxt
May 2017 40
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Tracing Lmod
bull A new feature of Lmod 744+bull module -T bull export LMOD_TRACING=yesbull Can trace loads and how restores work
May 2017 41
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
How to trace Lmod startup behavior
bull How do you trace module commands in etcprofiledshbull Could modify etcprofiledz99_StdEnvsh to turn on
LMOD_TRACING for a particular user UGHbull Install SHELL STARTUP DEBUG package instead
May 2017 42
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
SHELL STARTUP DEBUG Package
bull Install from shellstartupdebugsfnetbull Tracks startup behavior of etcprofiledshbull Allows for the setting of env vars before etcprofiledsh is sourcedbull Requires modifying etcbashrc
May 2017 43
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
SHELL STARTUP DEBUG (II)
bull export SHELL_STARTUP_DEBUG=1 (in ~initsh)bull Example output
etcprofileetcprofiledz00_lmodsh Time = 00770etcprofiledz99_StdEnvsh Time = 02067etcbashbashrc Time = 02338
Time = 03156
May 2017 44
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
SHELL STARTUP DEBUG and LMOD_TRACING
bull Put export LMOD_TRACING=yes in ~initshbull Track Lmod startup issues
running module --initial_load restoreUsing collection homeuserlmodddefaultSetting MODULEPATH to appsmfilesDarwinappsmfilesCoreLoading unix (fn appsmfilesCoreunixunixlua)Loading gcc (fn appsmfilesDarwingcc52lua)Loading StdEnv (fn appsmfilesCoreStdEnvlua)
May 2017 45
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Advanced Topics
bull mlbull Spider Cachebull Modulefiles can have propertiesbull New functions pushenv()bull SitePackagelua and hooks
May 2017 46
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
ml
bull I have trouble typing ``modulersquobull Needed a shortcut program not an aliasbull ml was bornbull ml -gt module listbull ml name -gt module load namebull ml ndasha b ndashc d -gt module unload a c module load b dbull ml av -gt module availbull ml spider -gt module spider
May 2017 47
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Spider Cache
bull Lmod has to know what modulefiles are in MODULEPATHbull It walks the directories in MODULEPATH every timebull Or you can have system spider cache to speed things upbull httpslmodreadthedocsioenlatest130_spider_cachehtml for
details
May 2017 48
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Spider Cache Advantages
bull The spider cache speeds up avail and spider greatlybull All system modulefiles have been read properties determinedbull Lua is quite fast and reading and interpreting a single filebull This is preferable to walking the directory tree and reading every modulebull Why is every module file read properties
May 2017 49
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Spider Cache Disadvantages
bull There is only one Keeping it up-to-datebull If Lmod sees a valid cache file it assumes it is correctbull Otherwise whats the pointbull Currently loads bypass cache but avail and spider depend on itbull Personal modules are not effected by system cache foo
May 2017 50
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Module Properties
bull TACC deployed Stampede with MIC acceleratorsbull Some modules will be ``MIC aware mkl fftw3 phdf5 bull Lmod will decorate these modules1) unixunix 3) ddtddt 5) mpich215 7) phdf5189 (m)
2) intel130 4) mklmkl () 6) petsc32 8) StdEnv
Where
(m) module is build natively for MIC
() module is build natively for MIC and offload to the MIC
add_property(archmic) -- gt phdf5 add_property(archmicoffload) -- gt mkl
May 2017 51
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Module Properties (II) Sticky
bull A module can be stickybull It requires ``--force to unload or purge
add_property(lmodsticky)
May 2017 52
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
pushenv()
bull Suppose youd like to set CC in the environmentbull setenv() wonrsquot workbull pushenv() will
setenv() pushenv()
$ module load gcc -gt CC=gcc CC=gcc
$ module load mpich -gt CC=mpicc CC=mpicc
$ module unload mpich -gt CC is unset CC=gcc
$ module unload gcc -gt CC is unset CC is unset
May 2017 53
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
SitePackagelua
bull Extra Functions for all modulefiles for a site needsbull Support for hooks like load_hook()bull See contribSitePackage for examples
May 2017 54
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Remote Debugging
bull No software over ten lines is bug freebull Lmod is no exceptionbull Bug reports are as easy as
bull module --config 2gt configlogbull module -D avail 2gt availlog
May 2017 55
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56
Conclusions Lmodbull Latest version httpsgithubcomTACCLmodgitbull Stable version httplmodsfnetbull Documentation httplmodreadthedocsorgbull Shell Startup Debug httpshellstartupdebugsfnetbull Mailing List lmod-userslistssourceforgenetbull Join here httpslistssourceforgenetlistslistinfolmod-users
May 2017 56