Modern Scientific Software Management using EasyBuild & cokehoste/EasyBuild_20170425... · Modern Scientific Software Management using EasyBuild & co - PRACE-VI-SEEM 2017 Spring School

ModernScientificSoftwareManagementusingEasyBuild&co

PRACE-VI-SEEM2017SpringSchool-SystemAdministrationTrackApril25th2017-TheCyprusInstitute

[email protected] http://hpcugent.github.io/easybuild/

http://www.ugent.be/hpc https://www.vscentrum.be

mailto:[email protected]

http://hpcugent.github.io/easybuild/

http://www.ugent.be/hpc

https://www.vscentrum.be/

ModernScientificSoftwareManagementusingEasyBuild&co-PRACE-VI-SEEM2017SpringSchoolKennethHoste(HPC-UGent)April25th2017-TheCyprusInstitute 2

• partofcentralITdepartmentofGhentUniversity(Belgium)

• centralisedscientificcomputingservices,training&support

• forresearchersofUGent,industry&knowledgeinstitutes

• memberofFlemishSupercomputerCentre(VSC)https://www.vscentrum.be/

• corevalues:empowerment-centralisation-automation-collaboration

HPC-UGent



• Masters&PhDinComputerSciencefromGhentUniversity(Belgium)

• joinedHPC-UGentteaminOctober2010

• maintasks:usersupport&training,softwareinstallations

• inheritedmaintenanceofEasyBuildin2011

• slowlyalsobecameleaddeveloper&releasemanager

• bigfanofloudmusic&FOSS(Free&OpenSourceSoftware)

[email protected](GitHub,IRC)-@kehoste(Twitter)

whoami



• focusisontoolsforHPCsystems,mostlyrelatedtousersupport

• inparticular,installing(scientific)softwareforusersofHPCclusters

• onlyactivelymaintained,well-documentedopensourcesoftwareprojects

• importantthingsIwillnotbetalkingabout(indetail):

• configurationmanagement(tips:Ansible,Warewulf,Quattor)

• security,monitoring,cloudservices,...

• MPI,performancetuning,accelerators,...

• thelatesthype(BigData,DeepLearning,...)

Scope


• Birds-eyeviewoftools&projects

• EasyBuild,Lmod,Singularity,ClusterShell,XALT/OGRT,OpenHPC

• IntroductiontoEasyBuild&Lmod

• Scope,motivation,terminology&features

• Installationandconfiguration

• Typicalusage

• Advancedtopics

Outline


• EasyBuild

• Lmod

• Singularity

• ClusterShell

• XALT&OGRT

• OpenHPC

Birds-eyeviewoftools&projects


• significantpartofusersupportservices

• traditionalpackagemanagersarenotsufficient

• installationshouldbedoneinanoptimisedway(fromsource)

• usersoftenlackrequiredknowledgetohandlethisthemselves(efficiently)

• scientificsoftwareinparticularcanbe...'challenging'

• littlecollaborationbetweenHPCsites,eventhoughthisisacommonburden

Usecase:installing(scientific)software

artworkonVSCBrENIACTier-1system


frameworkforbuilding&installing(scientific)softwareonHPCclusters


willbecoveredindetaillater...



• lotsof(scientific)softwareistypicallyprovidedonHPCclusters

• usuallyinnon-standardlocations,differentversions&variants,...

• usersneedtoquerywhatsoftwareisreadilyavailableandgeteasyaccesstoit

• wellestablishedsolutiononHPCclusters:environmentmodules

• traditionalenvironmentmodulestoolisoutdated,nolongeractivelymaintained...

Usecase:providingeasyaccesstosoftware


amodernenvironmentmodulestool

https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

willbecoveredindetaillater...

Lmod

https://www.tacc.utexas.edu/research-development/tacc-projects/lmod


• EasyBuild:getting(scientific)softwareinstalled

• Lmod:providingeasyaccesstosoftwareinstallations

• Singularity

• ClusterShell

• XALT&OGRT

• OpenHPC



• switchingtoHPCclusterusuallyrequiresuserstochangetheirworkflow

• differentenvironmenttoworkin(env.modules,etc.)

• requiredtoolsaremissingorinstalleddifferently

• maybealsodifferentversions,configuration,etc.

• confusingandtime-consumingforusers

• reproducingtheenvironmenttheyareusedtomayrequirealotofeffort

• usersmaybeunabletotakecareofitthemselves(lackpermissions&technicalexpertise,dependencyhell,...)

• supportteamsarebusy,maylackdomainknowledge

Usecase:sameworkflowonlocal&HPCsystems


• http://singularity.lbl.gov/-http://singularity.lbl.gov/user-guide

• containersolutionforHPCenvironments

• provides"mobilityofcompute":containerimagescanbeusedonany(Linux)system

• enablesreproduciblescience:containerimagesincludeeverythingthatisrequired

• dealswithsecurityconcernsofothercontainersolutionslikeDocker

• goodsupportforaccelerators(GPUs,...),MPI,Infiniband,etc.

Singularity

http://singularity.lbl.gov/

http://singularity.lbl.gov/user-guide


• developedbyHPCteamatLawrenceBerkeleyNationalLaboratory(LBNL)

• projectlead:GregoryM.Kurtzer(alsoknownfromCentOSLinux,Warewulf)

• firstpublicreleaseinApril2016,massiveuptakesince

• latestrelease:Singularityv2.2.1(Feb14th2017)

• downloadfrom:https://github.com/singularityware/singularity/releases

• tobeincludedinOpenHPCv1.3.1

Singularity

https://github.com/singularityware/singularity/releases

ModernScientificSoftwareManagementusingEasyBuild&co-PRACE-VI-SEEM2017SpringSchoolKennethHoste(HPC-UGent)April25th2017-TheCyprusInstitute

Singularity ArchitectureSingularity on the other hand is not designed around the idea of micro-service process isolation. Therefore it uses the smallest number of namespaces necessary to achieve its primary design goals.

This gives Singularity a much lighter footprint, greater performance potential and easier integration than container platforms that are designed for full isolation.

15

Singularityarchitecture

• similartoothercontainersolutions

• limiteduseofnamespaces

• loweroverheadthanDocker&co

• easytointegrate

(figurecourtesyofGregoryM.Kurtzer(LBNL)


• regularusersmustnotbeabletogainadminprivilegesinanyway...

• sensitiveissuebecauseofsharedfilesystems&lotsofresources

• noprivilegeescalationallowedinSingularity

• callinguserismaintainedwithinthecontainer

• ifyouwanttoberootinsidecontainer,youmustberootoutsidethecontainer

• norootowneddaemonprocess(asopposedtoDocker)

• containersarestartedasuserprocesses

SecurityconcernswithcontainersonHPCsystems


• containerimagesaresinglefiles

• easytodistribute&share

• veryefficientonparallelfilesystems

• self-containedcontainerimages:"itjustworks"

• containerimagescanbeusedonanyLinuxsystem

• BringYourOwnEnvironment(BYOE)

• userisfamiliarwithenvironment

encapsulatedincontainer

Mobilityofcompute,reproduciblescience&BYOESingularity: Bootstrapping the container$ sudo singularity bootstrap /tmp/Centos-7.img centos.def


• Singularityisdesignedfromthegrounduptointegrateeasilywithavailableresources

• containersarelaunchedviaresourcemanager,justlikeregularjobs

• nochangesrequiredtosystemconfiguration,itjustworks

• accesstoaccelerators(GPU,XeonPhi,...)iseasytocomeby

• someeffortisrequiredtouseMPIandleverageinterconnectslikeInfiniband

• requiresnecessarylibrariestobeavailableinthecontainerimage...

• sharedfilesystemscanbeboundintothecontainerthroughSingularityconfiguration

• supportsmultiplearchitectures(x86_64,POWER,AARCH64,...)

Integrationwithavailableresources


Singularity: Workflow./container.img

Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific Containers for Mobility of Compute (under review)

19

Singularityworkflow

adminprivilegesare(only)requiredtocreate

containerimages (andtoinstallSingularity)

(figurecourtesyofGregoryM.Kurtzer(LBNL)


• https://singularity-hub.org/

• createdbyVanessaSochat(Stanford'sResearchComputinggroup)

• platformtobuild&archiveSingularitycontainerimages

• designedtofacilitatereproducibleresearch,publications&peer-review

• integratedwithGitHubtotriggerbuildsandautomatecontainerisation

• containerimageshostedonSingularityHubcanbeuseddirectly:

singularity run shub://singularityhub/$USER/$CONTAINER:$TAG

SingularityHub

https://singularity-hub.org/

shub://singularityhub/$USER/$CONTAINER:$TAG


• benchmarkingexperimentsshowverylowoverheadcomparedtonativeexecution

• loweroverheadcomparedtoDocker&co

• sometimeseven(slightly)betterperformancethannative!(notescaleonY-axis,diff.issmall)

Singularity:performance

Containerized MPI Latency comparison over IB

● OpenMPI 2.0.1 with OSU Micro Benchmarks 5.3.2 http://mvapich.cse.ohio-state.edu/benchmarks/● Running over 40Gb/s Infiniband with Singularity v2.2

Note: Centos-7.2 uses libibverbs 1.1.8 and Centos-7.3 is 1.2.1, libmlx4 is (1.0.6 vs 1.2.1), among other OS differences which may account for the differences

Data provided by: Paulo Souza <[email protected]>

dataprovidedbyPauloSouza([email protected]),viaGregoryM.Kurtzer



• http://singularity.lbl.gov-https://github.com/singularityware/singularity

• presentations:

• FOSDEM'17(byMichaelBauer,UniversityofMichigan)

• https://fosdem.org/2017/schedule/event/singularity

• https://fosdem.org/2017/schedule/event/singularityhpc

• HPCACStanfordconference'17(byGregoryM.Kurtzer)

• http://insidehpc.com/2017/02/video-singularity-containers-science-reproducibility-hpc

• https://www.nextplatform.com/2017/04/10/singularity-containers-hpc-reproducibility-mobility

Singularity:moreinformation

http://singularity.lbl.gov/

https://github.com/singularityware/singularity

https://fosdem.org/2017/schedule/event/singularity/

https://fosdem.org/2017/schedule/event/singularityhpc/

http://insidehpc.com/2017/02/video-singularity-containers-science-reproducibility-hpc/

https://www.nextplatform.com/2017/04/10/singularity-containers-hpc-reproducibility-mobility/




• Singularity:containersforHPC

• ClusterShell

• XALT&OGRT

• OpenHPC



• HPCsystemadministratorsoftenneedtoruncommandsonremotesystems

• forexample:workernodesinanHPCcluster,loginnodes,managernodes,etc.

• poorsolution:for-looptorunacommandviaSSHoneachsystem

• veryslowforlargenumberofsystems(eventhoughitisembarrassinglyparallel...)

• hardtocomparecommandoutputs

• specifyinglistoftargetnodesmaybetricky(e.g.,excludingnodesthataredown)

• supportforrunningremotecommandsviaacustomPythonscriptwouldbeuseful

• pdsh:noPythonintegration,nointegratedCLI(pdsh,dshbak,pdcp),scalingissues,...

Usecase:runningcommandsonremotesystems


• website:http://cea-hpc.github.io/clustershell/

• documentation:http://clustershell.readthedocs.io/

• toolforrunningshellcommandsonremotesystems,inparallelandscalable

• supportformanagingnodesets,merging/diffingcommandoutputs,...

• canalsobeusedasaPythonlibrary(import ClusterShell)

• (originally)developedbyCEAHPCcenter(France)

• latestrelease:ClusterShellv1.7.3(Dec20th2016)

• downloadfrom:https://github.com/cea-hpc/clustershell/releases

ClusterShell

http://cea-hpc.github.io/clustershell/

http://clustershell.readthedocs.io/

https://github.com/cea-hpc/clustershell/releases


• toolforremoteexecutionofshellcommands,compatiblewithpdsh

• supportsSSH,RSH+customconnectionmodesusingPythonplugins

• targetnodesarespecifiedvianodesets(-w)orpredefined/dynamicnodegroups(-g)

• supportscopyingoffiles(inparallel,bothways)via-c/--copyand--rcopy

• smartdisplayofcommandresultsusing-b/-B/-Loptions

• integratedoutputgathering,sortingbynode,nodesetornodegroups

• degreeofparallelismcanbecontrolledvia-f/--fanout

• supportsconfigurable'treemode'forbetterscalability

ClusterShell:clushcommand


ClusterShell:nodesetandclubakcommands

• nodeset:toolformanipulationofnodesets,nodegroups

• expanding(-e),folding(-f),counting(-c)ofnodenames/sets/groups

• nodegroups:YAMLfiles,orviaexternalsources(e.g.SLURM'ssinfocommand)

• outputformatcanbetweakedaccordingtoneeds(-O)

• supportsnodesetarithmetic:joiningsets,excludingelements,etc.

• clubak:utilitytosummariseoutputandhighlightdifferences

• backwardscompatiblewithdshbakutilitythatisavailablewithpdsh

• leveragedbyclushwhen-b/-B/-Lisused


• nodesetmanipulation:

• commandexecution:

ClusterShellasPythonlibrary


ClusterShell:demo

# example usage of nodeset command$ nodeset -e 'node24[01-5]'node2401 node2402 node2403 node2404 node2405# predefined group of 200 workernodes in golett cluster$ nodeset -c @golett200

# output for 'module --version' on all golett workernodes (10 nodes in parallel)$ clush -B -f 10 -w @golett module --version---------------node[2400-2599] (200)---------------

Modules based on Lua: Version 6.6 2016-10-13 13:28 -05:00 by Robert McLay [email protected]



ClusterShell:moreinformation

• website:http://cea-hpc.github.io/clustershell/

• documentation:http://clustershell.readthedocs.io/

• wiki:https://github.com/cea-hpc/clustershell/wiki

• support:

• mailinglist:https://lists.sourceforge.net/lists/listinfo/clustershell-devel

• issuetracker:https://github.com/cea-hpc/clustershell/issues

• article:http://slashdot.org/journal/266980/A-Quick-Look-on-ClusterShell

• presentation:https://archive.fosdem.org/2016/schedule/event/hpc_bigdata_clustershell/

http://cea-hpc.github.io/clustershell/

http://clustershell.readthedocs.io/

https://github.com/cea-hpc/clustershell/wiki

https://lists.sourceforge.net/lists/listinfo/clustershell-devel

https://github.com/cea-hpc/clustershell/issues

http://slashdot.org/journal/266980/A-Quick-Look-on-ClusterShell

https://archive.fosdem.org/2016/schedule/event/hpc_bigdata_clustershell/





• ClusterShell:executingcommandsonremotesystems

• XALT&OGRT

• OpenHPC



• knowingwhich(scientific)softwareisbeingusedisimportant

• whichscientificsoftwareapplicationsconsumemostofavailablecomputetime?

• providedoptimisedinstallationsvsavailableviaOSvsuser-installed

• areprovidedsoftwareinstallationsactuallyusedlong-term?

• areoldsoftwareinstallationsstillbeingused?

• justcheckingwhichmodulesarebeingloadedisnotreliable

• loadingmodulesdoesnotimplythesoftwaretheyprovidewillactuallybeused

• whatifmultiplemodulesareloadedinjobs?

• usagetrackingshouldbeverylight-weightandshouldnevercauseanyproblems...

Usecase:trackingsoftwareusageonHPCsystems


• https://github.com/Fahey-McLay/xalt-http://xalt.readthedocs.io

• lateststablerelease:XALTv1.1.2(Jan4th2017),XALTv2isunderactivedevelopment

• tooltotrackwhichprograms/librariesusersareusingonHPCsystems

• successorofALTD(http://www.nersc.gov/users/software/programming-libraries/altd/)

• XALTv2isimplementedinC/C++,developedbyRobertMcLay(TACC)

• light-weight,transparent&robust

• hijackslinker(ld)tofingerprintexecutablessotheycanbetrackedindetail

• runtimetrackingbyXALTtriggeredviainit/finiinELFheaderofbinaries

XALT

https://github.com/Fahey-McLay/xalt

http://xalt.readthedocs.io

http://www.nersc.gov/users/software/programming-libraries/altd/


• https://github.com/georg-rath/ogrt

• latestrelease:OGRTv0.4.1(Sept28th2016)

• (another)tooltotrackwhichprograms/librariesusersareusingonHPCsystems

• implementedinC/Go,byGeorgRath(freelancer)

• light-weight(trackingallin-memory)andtransparent,resistanttooutsideinfluence

• noruntimedependencies,easytodeploy

• onlyworkswithdynamicbinaries,relieson$LD_PRELOAD

OGRT

https://github.com/georg-rath/ogrt


OGRTv0.4.1

• implementedinC/Go

• backends:SPLUNK,ElasticSearch

• heritage:trackmanysmalljobs/executables

• real-timedatacollection,senttoOGRTserver(viaUDP)

• trackingvia$LD_PRELOAD

• tracksallMPIranks

• usedat:IMBA,<privatecompany>

OGRTvsXALTside-by-sidecomparisonXALTv1.6.1-devel

(pre-v2)

• implementedinC/C++

• backends:database,file,syslog

• heritage:tracklargeMPIjobs(alsotracksnon-MPI)

• datatofile/syslog&digestednightlytoDB,canalsododirect-to-DBinreal-time

• trackingviaELFinit/finior$LD_PRELOAD• onlytracksMPIrank0(ignoresotherranks)

• usedat:TACC,NICS,Univ.Florida,KAUST,...

• commercialsupportbyEllexus


• quotebyRobertMcLay:"BigDatasucks!"

• easytocollectalotofdataonsoftwarebeingusedinjobs

• bydefaultallexecutablesaretracked,incl.standardtoolslikels,cd,cp,etc.

• alsocapturesalldefinedenvironmentvariables(typically~250perevent)

• ~17KBofdataperevent,easily>1GB/dayintotal...

• filteringisrequiredtofocusoneventsofinterest:

• onlyexecutables&environmentvariablesthatyoucareabout

• onlyonhoststhatreallymatter

• extracareshouldbetakenwithMPIjobs(e.g.,XALTonlytracksrank0)

(Big)DatacollectionwithXALT/OGRT


Collecteddataopenupwholerangeofpotentialusecases:

• targetedoutreach

• findoutwhichusersareusingoldsoftwareversions,buggylibraries,etc.

• verifyinghowsoftwareisbeingused

• dojobsreallyusetherequestedresources(cores,memory,...)?

• issoftwarebeingruninthecorrectenvironment(e.g.,usedMPIlibraries)?

• detailedstatisticsonwhichsoftwareisconsumingmostofavailablecomputetime

• wheretofocusoptimisationefforts(e.g.,awkusecasewithOGRT)

• canhelpwithtargetedbenchmarkingfornewsystems

UsecasesfordatacollectedwithXALT/OGRT


ExampleanalysisofdatacollectedwithXALT

top20executables onDarter(NICS) October'14-June'15

source: XALTpaper@HUST'15

https://doi.org/10.1145/2834996.2835000

https://doi.org/10.1145/2834996.2835000


XALT/OGRT:moreinformation

• GitHub:https://github.com/Fahey-McLay/xalt-https://github.com/georg-rath/ogrt

• documentation:http://xalt.readthedocs.io/en/latest-https://github.com/georg-rath/ogrt

• XALTpresentations

• FOSDEM'16:https://archive.fosdem.org/2016/schedule/event/hpc_bigdata_xalt/

• EUM'17:http://hpcugent.github.io/easybuild/files/EUM17/20170208-5_XALT.pdf

• OGRTpresentations:

• EUM'16:http://goo.gl/zbvChr-http://youtu.be/3l0eJq0nrOU

• FOSDEM'17:https://fosdem.org/2017/schedule/event/ogrt/

• jointBirds-of-a-FeathersessionatISC'17onXALT&OGRT

https://github.com/Fahey-McLay/xalt


http://xalt.readthedocs.io/en/latest


https://archive.fosdem.org/2016/schedule/event/hpc_bigdata_xalt/

http://hpcugent.github.io/easybuild/files/EUM17/20170208-5_XALT.pdf

http://goo.gl/zbvChr

http://youtu.be/3l0eJq0nrOU

https://fosdem.org/2017/schedule/event/ogrt/





• ClusterShell:executingcommandsonremotesystems

• XALT&OGRT:trackwhichsoftwareisbeingused

• OpenHPC



• referencecollectionofopen-sourceHPCsoftwarecomponents

• includesSLURM,Warewulf,Lmod,EasyBuild,ClusterShell,etc.

(soonalsoSingularity)

• supportsbothCentOS&SLES,x86_64andAARCH64(currentlyintechpreview)

• pre-builtpackagesforcompilers,MPI,libraries&toolsforHPC,etc.

• alsoincludesbestpractices,guidelinesforconfiguration,etc.

• intendstolowerthebarriertoentryforHPC

• initialreleaseatSC'15,latestreleasev1.3.0(March'17)

OpenHPC

OpenHPC Community BOF

Karl W. Schulz, IntelDavid Brayford, Leibniz Supercomputing Centre

November 16 ! Salt Lake City, Utah

http://openhpc.community


• website:http://openhpc.community

• GitHubrepository:https://github.com/openhpc/ohpc

• presentationatFOSDEM'16(Feb'16) OpenHPC:CommunityBuildingBlocksforHPCSystemshttps://fosdem.org/2016/schedule/event/hpc_bigdata_openhpc/

• paperatSystemProfessionalsWorkshop(Nov'16) ClusterComputingwithOpenHPC https://github.com/openhpc/ohpc/files/619162/HPCSYSPROS16_OpenHPC.pdf

OpenHPC:moreinformation

OpenHPC Community BOF

Karl W. Schulz, IntelDavid Brayford, Leibniz Supercomputing Centre

November 16 ! Salt Lake City, Utah

http://openhpc.community

http://openhpc.community/

https://github.com/openhpc/ohpc

https://fosdem.org/2016/schedule/event/hpc_bigdata_openhpc/

https://github.com/openhpc/ohpc/files/619162/HPCSYSPROS16_OpenHPC.pdf


• GettingScientificSoftwareInstalledwithEasyBuild&Lmod

• Context&motivationfortools&automation


• Installation&configuration

• Usage&workflow

• Advancedtopics

Outline


• resolvingproblemsthatoccurwhenusingtheHPCsystem(s)

- loginproblems,crashingjobs,thingsthat"stoppedworking",...

• answeringquestions,fromsimpletoverytechnical

• installing(scientific)softwaretools/libraries/applications

• helpingusersimprovetheirworkflow(notnecessarilybyrequest)

• training:Linuxbasics,OpenMP,MPI,Python,etc.

• performanceanalysisandoptimisationoflargescientificapplications

• consultancyservicesw.r.t.developingscientificsoftware

TasksforHPCusersupportteams


InstallingscientificsoftwareforusersinHPCsystems

• byuserrequest:newsoftware,versionupdates,morevariants,...

• usuallyona(sharedNFS)filesystemavailableoneveryworkernode

• specificallytargetedtotheHPCclusteritwillbeusedon

• builtfromsource(ifpossible)

• separateinstallationper(typeof)cluster

• highlyoptimisedforsystemarchitecture

• rebuildwhenupdatesforcompilers/librariesbecomeavailable

• installationstypicallyremainavailableduringlifetimeofsystem


"Pleaseinstall<software>ontheHPC?"Themostcommontypeofsupportrequestfromusersistoinstall(scientific)software.

• over25%ofsupportticketsatHPC-UGent

• consumes(way)morethan25%oftimeofHPC-UGentsupportteam

Installing(lotsof)scientificsoftwareis:

• error-prone,trial-and-error

• tedious,hardtogetright

• repetitive&boring(well...)

• time-consuming(hours,days,evenweeks)

• frustrating(e.g.,dependencyhell)

• sometimessimplynotworththeeffort...


CommonissueswithscientificsoftwareResearchersfocusonthesciencebehindthesoftwaretheyimplement,andcarelittleabouttools,buildprocedure,portability,...

Scientistsarenotsoftwaredevelopersorsysadmins(norshouldtheybe).

“Ifwewouldknowwhatwearedoing,itwouldnotbecalled‘research’.”

Thisresultsin:

• useofnon-standardbuildtools(orbrokenones)

• incompletebuildprocedure,e.g.,noconfigureorinstallstep

• interactiveinstallationscripts

• hardcodedparameters(compilers,libraries,paths,...)

• poor/outdated/missing/incorrectdocumentation


Primeexample:WRFWeatherResearchandForecastingModel(http://www.wrf-model.org)

• dozendependencies:netCDF(C,Fortran),HDF5,tcsh,JasPer,...

• knownissuesinlastreleaseare(only)documentedonwebsite

• nopatchfileprovided,infrequentbugfixreleases

• interactive'configure'script,optionschangebetweenversions

• resultingconfigurationfile'configure.wrf'needswork:fixhardcodedsettings(compilers,libraries,...),tweakingofoptions

• custom‘compile’script(wrapsaround‘make’) buildinginparallelisbrokenwithoutfixingtheMakefile

• noactualinstallationstep

http://www.wrf-model.org


Houston,wehaveaproblem...

InstallationofscientificsoftwareisatremendousproblemforHPCsitesallaroundtheworld.

• hugeburdenonHPCusersupportteams

• researchersloselotsoftime(waiting)

• sitestypicallyresorttocrudein-housescripting

• verylittlecollaborationamongHPCsites

http://geekand

poke.ty

pepad.com/geekand

poke/2010/05/how

-to-be

come-invaluable.htm

l

http://geekandpoke.typepad.com/geekandpoke/2010/05/how-to-become-invaluable.html


Whataboutexistingsoftwareinstallationtools?ExistingtoolsarenotwellsuitedtoscientificsoftwareandHPCsystems.

• packagemanagers:yum(RPMs),apt-get(.deb),...

• Homebrew(MacOSX),http://brew.sh/;Linuxbrew,http://brew.sh/linuxbrew/

• Portage(Gentoo),http://wiki.gentoo.org/wiki/Project:Portage

• pkgsrc(NetBSD&(alot)more),http://pkgsrc.org/

• Nix,http://nixos.org/nix;GNUGuix,https://www.gnu.org/s/guix

Commonproblems:

• usuallypoorsupportforold/multipleversionsand/ortohavebuildsside-by-side

• notflexibleenoughtodealwithidiosyncrasiesofscientificsoftware

• littlesupportforscientificsoftware,non-GCCcompilers,MPI

http://brew.sh/

http://brew.sh/linuxbrew/

http://wiki.gentoo.org/wiki/Project:Portage

http://pkgsrc.org/

http://nixos.org/nix

https://www.gnu.org/s/guix


• website:http://hpcugent.github.io/easybuild/

• documentation:http://easybuild.readthedocs.io

• frameworktobuild&installscientificsoftwareonHPCclusters

• implementedinPython2,leaddevelopmentbyHPC-UGent

• supportsdifferentcompilers&MPIlibraries,>1,150differentsoftwarepackages,...

• active&helpfulworldwidecommunity

http://easybuild.readthedocs.io


• (in-housedevelopmentatHPC-UGentsince2009)

• firstpublicreleaseinApril2012(EasyBuildv0.5)

• initialintentionwastogetfeedback,butacommunityemergedratherquickly...

• firststablereleaseinNovember2012(EasyBuildv1.0)

• frequentstablereleasessincethen(bothfeatureandbugfix)

• latest:EasyBuildv3.1.2(March20th2017)

• developmentishighlycommunity-driven(bugreports,featurerequests,contributions)

5yearsofEasyBuild


Featurehighlights(1)

• fullyautonomouslybuildingandinstalling(scientific)software

- automaticdependencyresolution

- automaticgenerationofmodulefiles(TclorLuasyntax)

• thoroughloggingofexecutedbuild/installprocedure

• archivingofbuildspecifications

• highlyconfigurable,viaconfigfiles/environment/commandline

• dynamicallyextendablewithadditionaleasyblocks,toolchains,etc.


Featurehighlights(2)

• supportforcustommodulenamingschemes(incl.hierarchical)

• transparencyviasupportfor'dryrun'installation

• comprehensivelytested:lotsofunittests,regressiontesting,...

• activelydeveloped,frequentstablereleases

• collaborationbetweenvariousHPCsites

• worldwidecommunity


WhatEasyBuildis(not)

EasyBuildis:

• notYABT(YetAnotherBuildTool);itdoesnotreplacetoolslikemake,itwrapsaroundthem

• notareplacementforyourfavouritepackagemanager

• notamagicsolutiontoallyour(softwareinstallation)problems...

• auniforminterfacethatwrapsaroundsoftwarebuildprocedures

• ahugetime-saver,byautomatingtedious/boring/repetitivetasks

• awaytoprovideaconsistentsoftwarestacktoyourusers

• anexpertsystemforsoftwareinstallationonHPCsystems

• aplatformforcollaborationwithHPCsitesworld-wide

• awaytoempoweruserstoself-managetheirsoftwarestackonHPCsystems


Supportedsoftware

• EasyBuildv3.1.2supportsinstallingover1,150softwaretools&applications

• includingCP2K,NAMD,NWChem,OpenFOAM,QuantumESPRESSO,WRF,WPS,...

• inaddition:hundredsofRlibraries,Pythonpackages,Perlmodules,...

• diversetoolchainsupport:

• compilers:GCC,Intel,Clang,PGI,IBMXL,Cray

• MPIlibraries:OpenMPI,IntelMPI,MPICH,MPICH2,MVAPICH2,...

• BLAS/LAPACKlibraries:IntelMKL,OpenBLAS,ATLAS,ACML,CrayLibSci,...

http://easybuild.readthedocs.io/en/latest/version-specific/Supported_software.html

http://easybuild.readthedocs.io/en/latest/version-specific/Supported_software.html


Terminology

• EasyBuildframework

- coreofEasyBuild:Pythonmodules&packages

- providessupportingfunctionalityforbuildingandinstallingsoftware

• easyblock

- aPythonmodulethatservesasabuildscript,'plugin'fortheEasyBuildframework

- implementsa(generic)softwarebuild/installprocedure

• easyconfigfile(*.eb):buildspecification;softwarename/version,compilertoolchain,etc.

• (compiler)toolchain:setofcompilerswithaccompanyinglibraries(MPI,BLAS/LAPACK,...)

• extensions:additionallibraries/packages/modulesforaparticularapplications(e.g.,Python,R)

http://easybuild.readthedocs.io/en/latest/Concepts_and_Terminology.html

http://easybuild.readthedocs.io/en/latest/Concepts_and_Terminology.html


InnumbersincludedinEasyBuildv3.1.2:

• about50kLoCofPython(incl.framework,easyblocks,andlogging/optionparsingsupport)

• about15kLoCmoreforunit/integrationtestsuites

• 1,182supportedsoftwareapplications(excl.extensions,softwareversions)

• 25differenttoolchaindefinitions(excl.toolchainversions)

• 181software-specificeasyblocks,30genericeasyblocks

• 6,718easyconfigfiles

• differentsoftwareversions,variants,usingdifferenttoolchains,...


Providing(easy)accesstosoftwareinstallations

• lotsof(scientific)softwareistypicallyinstalledonHPCclusters• potentiallywiderangeofscientificdomains

• multipledifferentversions,builds,variants,...

• installedwithdifferentgenerationsofcompilers/libraries

• usuallyinanon-standardlocation(sharedfilesystem)

• environmentneedstobemodifiedtousethesesoftwareinstallations

• $PATH,$LD_LIBRARY_PATH,etc.+customsoftware-specificenvironmentvariables

• howcanweeasilyletusersaccessthesesoftwareinstallations?


Environmentmodules• wellestablishedsolutioninHPCcommunity

• shell-agnostic'scripts'tosetupenvironment

• intuitiveuserinterfaceviamodulecommand

• usedonmostHPCsystems(>80%1),sincemid90's

• Tcl-basedenvironmentmodulessystemisubiquitous,but:

• lastreleaseofTcl/Cimplementation(v3.2.10)wasDec2012

• pureTclversionisactivelymaintained,but(alot)lessused

• lacksfeaturessupportedbymorerecentimplementations...

(1)http://hpcugent.github.io/easybuild/files/SC15_BoF_Getting_Scientific_Software_Installed.pdf

http://hpcugent.github.io/easybuild/files/SC15_BoF_Getting_Scientific_Software_Installed.pdf


Environmentmodules:userinterface

• checkingwhichmodulesareavailableisdoneusingmodule avail

• toupdatetheenvironmentforaparticularsoftwareinstallation,

amoduleshouldbeloadedusingmodule load

• anoverviewofcurrentlyloadedmodulescanbeobtainedwithmodule list

• reversingthechangestotheenvironmentcanbedonewithmodule unload

• resettingtheenvironmentbyunloadingallloadedmodulescanbedone

usingmodule purge


$ which gcc

/usr/bin/gcc

$ module avail GCC

--------------------- /apps/modules/all ---------------------

GCC/4.7.4 GCC/4.8.5 GCC/4.9.4 GCC/5.1.0 GCC/5.2.0 GCC/5.3.0

$ module load GCC/5.3.0

$ which gcc

/apps/software/GCC/5.3.0/bin/gcc

62

Environmentmodules:userinterface


• website:https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

• documentation:http://lmod.readthedocs.io/

• latest:Lmodv7.4.4(April17th2017)-https://github.com/TACC/Lmod/releases

• implementedinLua,leaddevelopmentbydr.RobertMcLay(TACC)

• mostlydrop-inreplacementforTcl-basedenvironmentmodulestool('Tmod')

• activelymaintained,(very)frequentupdates,vibrant&helpfulcommunity

• slowlytakingoverthe(HPC)world...

Lmod:amodernenvironmentmodulestool

https://github.com/TACC/Lmod/releases


Lmod:featurehighlights(1)• samemoduleuserinterfaceasTmod

• consumesmodulefileswrittenineitherTclorLua,mixingofbothissupported

• modulefilesarecachedtomakeuserinterfacemoreresponsive

• Lmod'sspidercachemustbekeptup-to-date!(cronjob)

• alsoprovidesmlshorthandcommand('moudle'is(too)easytomistype)

• mlisthesameasmodule list• ml fooisthesameasmodule load foo• ml avisthesameasmodule available• ml -fooisthesameasmodule unload foo


Lmod:featurehighlights(2)

• someminorfunctionaldifferenceswithTmod:

• onenamerule:onlyonesinglemodulecanbeloadedforaparticularsoftware

• canalsobeenforcedinTmodvia'conflict'statementsinmodulefiles

• Lmodautomaticallyunloadspreviouslyloadedmoduleonnameclash(bydefault)

• correctorderingofsoftwareversions(5.6isolderthan5.10)

• propersupportforhierarchicalmodules

• cansearchthroughallexistingmodulesvia'module spider',eveniftheyarenot(yet)availableforloading


Lmod:featurehighlights(3)• supportforadditionalfeatures,including:

• definingmodulefamilies(e.g.MPI),onlyonemodulecanbeloadedperfamily

• moduleproperties,whichareshowninoutputof'module avail'

• stickymodules,whichdonotgetunloadedon'module purge'(unlessforced)

• user-definedmodulecollectionsvia'module save'and'modulerestore'

• deprecatingmodules,usercangetacustommessagewhenamoduleisloaded

• hookstocustomiseLmod'sbehaviour,e.g.,totrackwhichmodulesgetloaded

• easydebuggingofLmoditselfvia'-D'optiontomodule/mlcommands

• ...


Communityaspects• EasyBuild&Lmodcommunitieshavebeengrowingrapidlythelastcoupleofyears

• hundredsofHPCsitesworldwide,largeandsmall

• verywelcoming&supportivetonewcomers

• significantoverlapbetweenEasyBuild&Lmodcommunities

• activemailinglists:

• EasyBuild:https://lists.ugent.be/wws/info/easybuild

• Lmod:https://sourceforge.net/p/lmod/mailman/lmod-users/

• EasyBuildalsohasactiveIRCandSlackchannels

• usersalsostartcontributing:bugreports,featurerequests,codecontributions,...

https://lists.ugent.be/wws/info/easybuild

https://sourceforge.net/p/lmod/mailman/lmod-users/


EasyBuildmailinglist(subscribers)

(2013) +25

(2014) +30

(2015) +60

(2016) +55


EasyBuildmailinglist(traffic)

0

200

400

600

800

1000

1200

2012 2013 2014 2015 2016 2017

#messages/year


0

20

40

60

80

100

120

140

Jan-12 Jul-12 Jan-13 Jul-13 Jan-14 Jul-14 Jan-15 Jul-15 Jan-16 Jul-16 Jan-17 Jul-17

#un

ique

contrib

utors

framework

easyblocks

easyconfigs

70

CodecontributionstoEasyBuild

(2013) +20

(2014) +20

(2015) +30

(2016) +40






• Usage&workflow

• Advancedtopics

Outline


EasyBuildrequirements

• focusisonLinuxx86_64HPCsystems

• RedHat-based(CentOS,SL,...),alsoonDebianandSuSEderivatives

• also(kindof)worksonmacOS,butnotatargetplatform

• basicWindowssupportvianewWindowsSubsystemforLinux(WSL)

• ongoingeffortstoenhancesupportonAARCH64(ARM)andPowerLinux

• Python2.6ormorerecentPython2,incl.setuptools(noPython3supportyet)

• systemC/C++compiler(GCC),usedtokickstartinstallationoftoolchain

• anenvironmentmodulestool(Lmodishighlyrecommended!)

http://easybuild.readthedocs.io/en/latest/Installation.html#requirements

http://easybuild.readthedocs.io/en/latest/Installation.html#requirements


InstallingEasyBuild

• installingPythonpackagescanbemessy,cfr.plethoraofPythoninstallationtools

• bootstrapscriptforEasyBuildwascreatedoutoffrustration

• standardinstallationtoolslikeeasy_install,pip,etc.worktoo

• packagingeffortsunderway(seeOpenHPC;WIPinFedora)

# download bootstrap script from # https://raw.github.com/hpcugent/easybuild-framework/develop/easybuild/scripts/bootstrap_eb.py$ python bootstrap eb.py $PREFIX $ module use $PREFIX/modules/all $ module load EasyBuild

http://easybuild.readthedocs.io/en/latest/Installation.html

http://easybuild.readthedocs.io/en/latest/Installation.html


UpdatingEasyBuild

• newreleasesarefullybackwards-compatible(exceptfornewmajorversions)

• toupdateEasyBuild:

• re-bootstraptoobtainamoduleforlatestEasyBuildrelease

• oruse'eb --install-latest-eb-release'toinstallmoduleforlatestrelease

• installaspecificEasyBuildversionwithEasyBuildusinganeasyconfigfilehttps://github.com/hpcugent/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/e/EasyBuild

http://easybuild.readthedocs.io/en/latest/Installation.html#updating-an-existing-easybuild-installation

https://github.com/hpcugent/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/e/EasyBuild

http://easybuild.readthedocs.io/en/latest/Installation.html#updating-an-existing-easybuild-installation


ConfiguringEasyBuild(1)

YoushouldconfigureEasyBuildtoyourpreferences,via:

• configurationfile(s):key-valuelines,textfiles(e.g.,prefix=/tmp)

• environmentvariables(e.g.,$EASYBUILD_PREFIXsetto/tmp)

• commandlineparameters(e.g., --prefix=/tmp)

Eachconfigurationoptioncanbesetoneachoftheseconfigurationlevels.

Priorityamongdifferentoptions:cmdline,envvars,configfile.Forexample:

• --prefixoverrules$EASYBUILD_PREFIX

• $EASYBUILD_PREFIXoverrulesprefixinconfigurationfile

http://easybuild.readthedocs.io/en/latest/Configuration.html



ConfiguringEasyBuild(2)

Bydefault,EasyBuildwill(ab)use$HOME/.local/easybuild. Use'eb --show-config'togetanoverviewofthecurrentconfiguration.


$ EASYBUILD_PREFIX=/tmp eb --buildpath /dev/shm --show-config## Current EasyBuild configuration# (C: command line argument, D: default value, E: environment variable, F: configuration file)#buildpath (C) = /dev/shminstallpath (E) = /tmppackagepath (E) = /tmp/packagesprefix (E) = /tmprepositorypath (E) = /tmp/ebfiles_reporobot-paths (D) = /Users/kehoste/work/easybuild-easyconfigs/easybuild/easyconfigssourcepath (E) = /tmp/sources



InstallingLmod

• requirements:

• Lua5.1ornewer,incl.luaposixandluafilesystemlibraries

• luajsonandluatermarealsorecommended(butnotstrictlyrequired)

• Lmodinstallationprocedureisstandardconfigure-make-make install

• profilescriptsareincluded,requiredtodefinemodule/mlcommandsonlogin

• typicallyinstalledsystem-wideasanOSpackage,withsite-specificconfiguration

• .specfilefor(our)Lmodisavailableathttps://github.com/hpcugent/Lmod-UGent

http://lmod.readthedocs.io/en/latest/030_installing.htmlhttps://github.com/TACC/Lmod/blob/master/INSTALL

https://github.com/hpcugent/Lmod-UGent

http://lmod.readthedocs.io/en/latest/030_installing.html

https://github.com/TACC/Lmod/blob/master/INSTALL


ConfiguringLmod(1)

• defaultLmodbehaviourcanbeconfiguredatbuildtimeusingconfigureoptions

• canalsobetweakedatruntimeusing$LMOD_*environmentvariables

• someaspectsareconfiguredvialmodrc.luafile

• locationofLmodspidercache,knownmoduleproperties,...

• hookscanbeimplementedinSitePackage.luafile

• activeLmodconfigurationcanbequeriedvia'module --config'

http://lmod.readthedocs.io/en/latest/155_lmodrc.html

http://lmod.readthedocs.io/en/latest/170_hooks.html

http://lmod.readthedocs.io/en/latest/155_lmodrc.html

http://lmod.readthedocs.io/en/latest/170_hooks.html


ConfiguringLmod(2)

• somenoteworthyconfigurationoptions:

• --with-redirect=yes:redirectmessagestostdout(otherthanwarnings/errors)

• --with-disableNameAutoSwap=yes:produceanerrorratherthan automaticallyunloadingmoduleonnameclash(onenamerule)

• --with-shortTime=<N>:withlargeN,avoidbuildingofuserspidercachefile

• --with-pinVersions=yes:alwayspinmoduleversionsincollections

• --with-cachedLoads=yes:alsousespidercachefor'module load'


Lmod:spidercache

• makingLmoduseanup-to-datesystemspidercachefileisveryimportant

• requiredforresponsive'module'/'ml'

• locationofsystemspidercachecanbecustomised:

• viaconfigureoptions:--with-spiderCacheDir,...

• viaactivelmodrc.luaconfigurationfile(seescDescriptTentry)

• verifyconfigurationvia'ml --config'

• useupdate_lmod_system_cache_filestoupdatesystemspidercacheviacron

• avoidbuildingofuserspidercachebyspecifyinglargevalueto--with-shortTime

http://lmod.readthedocs.io/en/latest/130_spider_cache.html

http://lmod.readthedocs.io/en/latest/130_spider_cache.html






• Usage&workflow

• Advancedtopics

Outline


Basicusage

• specifysoftwarename/versionandtoolchainto'eb'command

• commonlyviaeasyconfigfilename(s):

eb GCC-4.9.2.eb Clang-3.6.0-GCC-4.9.2.eb

• checkwhetherrequiredtoolchain&dependenciesareavailableusing--dry-run/-D:

eb Python-2.7.13-intel-2017a.eb -D

• enabledependencyresolutionvia--robot/-r:

eb WRF-3.8.0-intel-2016b-dmpar.eb -dr

http://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html

http://easybuild.readthedocs.io/en/latest/Typical_workflow_example_with_WRF.html

http://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html

http://easybuild.readthedocs.io/en/latest/Typical_workflow_example_with_WRF.html


83

Step-wiseinstallationprocedure

IV: unpack sources

V: apply patches

VI: prepare

VII: configure

VIII: build

IX: test

X: install

XI: extensions

XII: sanity check

XIII: cleanup

XIV: env. module

III: check readiness XV: permissions

II: fetch sources XVI: packaging

I: parse easyconfig XVII: test cases

EasyBuildperformsastep-wiseinstallationprocedureforeachsoftware

• downloadsources(besteffort)

• setupbuilddirectory&environment

• unpacksources(&applypatches)

• loadmodulesfortoolchain&deps

• definetoolchain-relatedenvvars($CC,$CFLAGS,...)

• configure,build,(test),install,(extensions)

• performasimplesanitycheckoninstallation

• generatemodulefile

eachstepcanbecustomisedviaeasyconfigparametersoraneasyblock


Exampleoutputofusing'eb'command$ eb GCC-4.9.3.eb == temporary log file in case of crash /tmp/eb-GyvPHx/easybuild-U1TkEI.log == processing EasyBuild easyconfig GCC-4.9.3.eb == building and installing GCC/4.9.3... == fetching files... == creating build dir, resetting environment... == unpacking... == patching... == preparing... == configuring... == building... == testing... == installing... == taking care of extensions... == postprocessing... == sanity checking... == cleaning up... == creating module... == permissions... == packaging... == COMPLETED: Installation ended successfully == Results of the build can be found in the log file /opt/easybuild/software/GCC/4... == Build succeeded for 1 out of 1 == Temporary log file(s) /tmp/eb-GyvPHx/easybuild-U1TkEI.log* have been removed. == Temporary directory /tmp/eb-GyvPHx has been removed.


Easyconfigfilesasbuildspecifications

• simpletextfilesincludingasetofeasyconfigparameters(inPythonsyntax)

• somearemandatory:softwarename/version,toolchain,metadata(homepage,descr.)

• othercommonlyusedparameters:

• easyblocktouse

• listofsources&patches

• (build)dependencies

• optionsforconfigure/build/installcommands

• filesanddirectoriesthatshouldbepresent(sanitycheck)

http://easybuild.readthedocs.io/en/latest/Writing_easyconfig_files.html



Exampleeasyconfigfile

name = 'WRF'version = '3.8.0'

buildtype = 'dmpar' # custom parameter for WRFversionsuffix = '-' + buildtype # part of module name

homepage = 'http://www.wrf-model.org'description = "Weather Research and Forecasting (WRF) Model"

toolchain = {'name': 'intel', 'version': '2016b'}

source_urls = ['http://www.mmm.ucar.edu/wrf/src/']sources = ['%(name)sV%(version_major_minor)s.TAR.gz']patches = ['WRF-%(version)s_known_problems.patch']

builddependencies = [('tcsh', '6.20.00')]dependencies = [ ('JasPer', '2.0.10'), ('netCDF', '4.4.1'), ('netCDF-Fortran', '4.4.4'),]

softwarenameandversion

buildvariant(specifictoWRF)('dmpar':distributed,MPI)

softwaremetadata

sources&patches

toolchainname&version

listof(build)dependenciesnote:allversionsarefixed!

noeasyblockspecified,whichimpliesusingasoftware-specificeasyblock(EB_WRF)






• Typicalusage&workflow

• Advancedtopics

Outline


Logfiles

• EasyBuildthoroughlylogsexecutedinstallationprocedure

• activeEasyBuildconfiguration

• easyconfigfilethatwasused

• modulesthatwereloaded+resultingchangestoenvironment

• definedenvironmentvariables

• outputofexecutedcommands

• informativelogmessagesproducedbyeasyblock

• logisfileiscopiedtosoftwareinstallationdirectoryforfuturereference

• canbeusedtodebugbuildproblemsorseehowinstallationwasperformedexactly

http://easybuild.readthedocs.io/en/latest/Logfiles.html

http://easybuild.readthedocs.io/en/latest/Logfiles.html


Logfiles:example== 2016-04-24 13:34:31,906 main.EB_HPL INFO This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on host example....== 2016-04-24 13:34:35,503 main.EB_HPL INFO configuring...== 2016-04-24 13:34:48,817 main.EB_HPL INFO Starting configure step...== 2016-04-24 13:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure...== 2016-04-24 13:34:48,823 main.run DEBUG run_cmd: running cmd /bin/bash make_generic (in /tmp/user/easybuild_build/HPL/2.0/goolf-1.4.10/hpl-2.0/setup)== 2016-04-24 13:34:48,823 main.run DEBUG run_cmd: Command output will be logged to /tmp/easybuild-W85p4r/easybuild-run_cmd-XoJwMY.log== 2016-04-24 13:34:48,849 main.run INFO cmd "/bin/bash make_generic" exited with exitcode 0 and output:...


Addingsupportforadditionalsoftware

• foreachsoftwareinstallation,aneasyconfigfileisrequired

• defineseasyconfigparametersthatspecifytoEasyBuildwhattoinstall,andhow

• existingeasyconfigfilescanserveasexamples

• forversionortoolchainupdates,atweakedeasyconfigcanbegeneratedviaeb --try-*

• seehttp://easybuild.readthedocs.io/en/latest/Writing_easyconfig_files.html

• for'standard'installationprocedures,agenericeasyblockcanbeused

• genericinstallationcanbecontrolledwhereneededviaeasyconfigparameters

• forcustominstallationprocedures,asoftware-specificeasyblockismustbeimplemented

• seehttp://easybuild.readthedocs.io/en/latest/Implementing-easyblocks.html


http://easybuild.readthedocs.io/en/latest/Implementing-easyblocks.html


Commongenericeasyblocks

• ConfigureMake

standard'./configure'-'make'-'make install'installationprocedure

• CMakeMake

sameasConfigureMake,butusingCMakeforconfiguring

• PythonPackage

installingPythonpackages('pythonsetup.pyinstall','pip install',...)

• MakeCp no(standard)configurationstep,buildwith'make',installbycopyingbinaries/libraries

• Tarball:justunpacksourcesandcopyeverythingtoinstallationdirectory

• Binary:runbinaryinstaller(specifiedvia'install_cmd'easyconfigparameter)

http://easybuild.readthedocs.io/en/latest/version-specific/generic_easyblocks.html

http://easybuild.readthedocs.io/en/latest/version-specific/generic_easyblocks.html


Easyconfigfilesvseasyblocks

• thinlinebetweenusingcustomeasyblockand'fat'easyconfigwithgenericeasyblock

• customeasyblocksare"doonceandforget",centralsolutiontobuildpeculiarities

• reasonstoconsiderimplementingasoftware-specificeasyblockinclude:

• 'critical'valuesforeasyconfigparametersrequiredtomakeinstallationsucceed

• toolchain-specificaspectsofthebuildandinstallationprocedure(e.g.,configureoptions)

• interactivecommandsthatneedtoberun

• custom(configure)optionsfordependencies

• havingtocreateoradjustspecific(configuration)files

• 'hackish'usageofagenericeasyblock




Commontoolchains

• intelandfoss1toolchainsaremostcommonlyusedinEasyBuildcommunity

• helpstofocuseffortsofHPCsitesusingoneorbothofthesetoolchains

• updatedtwiceayear,clearversioningscheme(2016b,2017a,...)

• latestversion:

• foss/2017a binutils2.27,GCC6.3.0,OpenMPI2.0.2,OpenBLAS0.2.19,LAPACK3.7.0,FFTW3.3.6

• intel/2017a binutils2.27+GCC6.3.0asbaseversion2017.1.132ofIntelcompilers,IntelMPIandIntelMKL

http://easybuild.readthedocs.io/en/latest/Common-toolchains.html

(1)FOSS:FreeandOpenSourceSoftware

http://easybuild.readthedocs.io/en/latest/Common-toolchains.html


Transparencyofperformedinstallprocedure

• 'eb --extended-dry-run','eb -x'revealsplannedinstallationprocedure

• runsinamatterofseconds

• showscommandsthatwillbeexecuted,buildenvironment,generatedmodulefile,...

• anyerrorsthatoccurinusedeasyblockareignoredbutclearlyreported

• not100%accuratesinceeasyblockmayrequirecertainfilestobepresent,etc.

• veryusefulwhendebuggingeasyblocks,instantfeedbackasafirstpass

• implementationmotivatedbyrequestsfromthecommunity

• helpstoavoidimpressionthatEasyBuildisamagicblackboxforinstallingsoftware

http://easybuild.readthedocs.io/en/latest/Extended_dry_run.html



$ eb WRF-3.8.0-intel-2016b-dmpar.eb -x

== temporary log file in case of crash /tmp/eb-Dh1wOp/easybuild-0bu9u9.log

== processing EasyBuild easyconfig /home/example/eb/easybuild-easyconfigs/easybuild/easyconfigs/w/WRF/WRF-3.8.0-intel-2016b-dmpar.eb

...

*** DRY RUN using 'EB_WRF' easyblock (easybuild.easyblocks.wrf @ /home/example/eb/easybuild-easyblocks/easybuild/easyblocks/w/wrf.py) ***

== building and installing WRF/3.8.0-intel-2016b-dmpar...fetching files... [DRY RUN]

[fetch_step method]Available download URLs for sources/patches: * http://www2.mmm.ucar.edu/wrf/src//$source * http://www.mmm.ucar.edu/wrf/src//$source

List of sources: * WRFV3.8.0.TAR.gz will be downloaded to /home/example/eb/sources/w/WRF/WRFV3.8.0.TAR.gz...

Exampleoutputof--extended-dry-run(1)



...

building... [DRY RUN]

[build_step method] running command "tcsh ./compile -j 4 wrf" (in /home/example/eb/software/WRF/3.8.0-intel-2016b-dmpar/WRF-3.8.0) running command "tcsh ./compile -j 4 em_real" (in /home/example/eb/software/WRF/3.8.0-intel-2016b-dmpar/WRF-3.8.0) running command "tcsh ./compile -j 4 em_b_wave" (in /home/example/eb/software/WRF/3.8.0-intel-2016b-dmpar/WRF-3.8.0)...

[sanity_check_step method]Sanity check paths - file ['files'] * WRFV3/main/libwrflib.a * WRFV3/main/real.exe * WRFV3/main/wrf.exeSanity check paths - (non-empty) directory ['dirs'] * WRFV3/main * WRFV3/runSanity check commands (none)



...

[make_module_step method]Generating module file /home/example/eb/modules/all/WRF/3.8.0-intel-2016b-dmpar, with contents:

#%Module proc ModulesHelp { } { puts stderr { The Weather Research and Forecasting (WRF) Model } } module-whatis {Description: WRF - Homepage: http://www.wrf-model.org}

set root /home/example/eb/software/WRF/3.8.0-intel-2016b-dmpar

conflict WRF

if { ![ is-loaded intel/2016b ] } { module load intel/2016b }

if { ![ is-loaded JasPer/1.900.1-intel-2016b ] } { module load JasPer/1.900.1-intel-2016b }

if { ![ is-loaded netCDF/4.4.1-intel-2016b ] } {



Usingacustommodulenamingscheme• acoupleofdifferentmodulenamingschemesareincludedinEasyBuild

• see--avail-module-naming-schemes

• specifyactivemodulenamingschemevia--module-naming-scheme

• default:EasyBuildMNS(<name>/<version>-<toolchain>-<versionsuffix>)

• youcanimplementyourownmodulenamingschemerelativelyeasily

• specifyhowtocomposemodulenameusingprovidedmetadata

• viaPythonmodulethatdefinescustomderivativeclassofModuleNamingScheme

• makeEasyBuildawareofitvia--include-module-naming-schemes

• decouplenamingofinstalldirsvsmodulesvia--fixed-installdir-naming-scheme


Flatvshierarchicalmodulenamingscheme• flatmodulenamingscheme:

• allexistingmodulesarealwaysavailableforloading

• downsides:

• long(confusing)modulenamestodistinguishbuildwithdifferenttoolchains

• userscaneasilyshootthemselvesinthefootby(trying)toloadincompatiblemodules

• hierarchicalmodulenamingscheme:

• modulesareorganisedinatree-likefashion

• a'core'modulemustbeloadedfirsttomakemore(compatible)modulesavailable

• typicallyusedwithdifferenthierarchylevels:Core-Compiler-MPI


Hierarchicalmodulenamingschemeindetail

• initially,onlyCoremodulesareavailableforloading

• othermodulesareonlyvisiblevia'module spider'

OpenMPI/2.1.0OpenMPI/2.1.0OpenMPI/1.10.6

GCC/5.3.0 GCC/6.1.0

OpenMPI/1.10.6

FFTW/3.3.4 FFTW/3.3.5 FFTW/3.3.6 FFTW/3.3.4 FFTW/3.3.5 FFTW/3.3.6

(loaded)

(available)

(notavailable)

Corelevel

Compilerlevel (compiler-dependent)

MPIlevel (MPI-dependent)

legend



• Coremodulesmayextend$MODULEPATHwithanadditionallocation

• loadingaCoremodulemaymakemoremodulesavailable

• inthisexample,loadingaGCCmodulemakesOpenMPImodulesavailable


GCC/5.3.0 GCC/6.1.0

OpenMPI/1.10.6


(loaded)

(available)

(notavailable)

Corelevel



legend




GCC/5.3.0 GCC/6.1.0

OpenMPI/1.10.6


(loaded)

(available)

(notavailable)

Corelevel



legend

• evenmoremodulesmaybemadeavailablebyloadingothermodules

• forexample,loadinganOpenMPImodulesrevealsMPI-dependentmodules


IntegrationwithGitHub

• accesstoGitHubstraightfromEasyBuildcommandline(!)

• 'eb --from-pr'topullineasyconfigsfromaspecificpullrequest

• --upload-test-reporttoaddcommentwithtestreportintoapullrequest

• 'eb --new-pr'tocreateanewpullrequestforcontributing(changesto)easyconfigs

• 'eb --update-pr'toaddupdatestoexistingpullrequests

• significantlylowersthebarforcontributing,noneedtobefamiliarwithgit

• alsoturnsouttobeahugetimesaverforexperiencedcontributors

• fornowonlysupportedforeasyconfigfiles,soonalsoforeasyblocks&framework

http://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html

http://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html


Otherfeaturesthatwerenotcovered...

• lettingusersmanagetheirsoftwarestackontopofcentrallyprovidedmodules

• installinghiddenmodules,hidingcertaindependencies&toolchains

• supportforusingRPATHlinking

• partialinstallations:only(re)generatemodulefile,installadditionalextensions

• dynamicallyextendingEasyBuildvia--include-*

• submittinginstallationsasjobstoanHPCclustervia--job

• creatingpackages(RPMs,...)forsoftwareinstallationsdonewithEasyBuild

• usingEasyBuildonCraysystems,integrationwithCrayProgrammingEnvironment




ContributingtoEasyBuild&Lmod

BothEasyBuildandLmodhaveimprovedsignificantlythankstotheircommunities.

Youtoocancontributeback,by:

• sendingfeedback

• reportingbugs

• joiningthediscussion(mailinglists,EasyBuildconfcalls)

• sharingsuggestionsandideasforchanges&additionalfeatures

• contributingeasyconfigs,enhancingeasyblocks,addingsupportfornewsoftware...

• extending&enhancingdocumentation

http://easybuild.readthedocs.io/en/latest/Contributing.html

http://easybuild.readthedocs.io/en/latest/Contributing.html


vs

• Spack(http://spack.io/)isapackagemanagerfocusedonscientificsoftwareandHPC

• leaddevelopmentbyToddGamblin(LLNL)

• significantgrowthinuserbaseinrecentmonths

• similaryetdifferentapproachcomparedEasyBuild

• veryflexiblecommandlineinterfacew.r.t.specifyingwhattoinstall

• concretisespartialspecification,dependencyversionsdon'tneedtobefixed

• focusismoreonsoftwaredevelopers

• someexperiencewithGit&Pythonisassumed/required

http://spack.io/


PapersonEasyBuild(&Lmod)

ModernScientificSoftwareManagementUsingEasyBuildandLmod

MarkusGeimer(JSC),KennethHoste(HPC-UGent),RobertMcLay(TACC)

http://hpcugent.github.io/easybuild/files/hust14_paper.pdf

MakingScientificSoftwareInstallationReproducibleOnCraySystemsUsingEasyBuild

PetarForai(IMP),GuilhermePeretti-Pezzi(CSCS),KennethHoste(HPC-UGent)

https://cug.org/proceedings/cug2016_proceedings/includes/files/pap145.pdf

ScientificSoftwareManagementinRealLife:DeploymentofEasyBuildonaLargeScaleSystem

DamianAlvarez,AlanO’Cais,MarkusGeimer(JSC),KennethHoste(HPC-UGent)

http://hpcugent.github.io/easybuild/files/eb-jsc-hust16.pdf

http://hpcugent.github.io/easybuild/files/hust14_paper.pdf

https://cug.org/proceedings/cug2016_proceedings/includes/files/pap145.pdf

http://hpcugent.github.io/easybuild/files/eb-jsc-hust16.pdf

ModernScientificSoftwareManagementusingEasyBuild&co

PRACE-VI-SEEM2017SpringSchool-SystemAdministrationTrackApril25th2017-TheCyprusInstitute

[email protected] http://hpcugent.github.io/easybuild/

http://www.ugent.be/hpc https://www.vscentrum.be



http://www.ugent.be/hpc


Documents

Modern Scientific Software Management using EasyBuild & cokehoste/EasyBuild_20170425... · Modern Scientific Software Management using EasyBuild & co - PRACE-VI-SEEM 2017 Spring School