Upload
dominh
View
222
Download
1
Embed Size (px)
Citation preview
Setup for the workshop1. Get a user ID and password paper (provided in class):
##:**********
2. Access to local computer (replace “##” and “___” with appropriate values, “___” is provided in class):a. User name: csuser## (ex.: csuser99)b. Password: ___@[S## (ex.: sec@[S99)
3. Slides:
http://tinyurl.com/cq-intro-arc-20170928
3
Outline
● Introduction to ARC○ Laptop vs HPC Cluster
● Moving Data○ With Globus
● Software Development○ Types of Parallelism○ Scientific Software and Modules○ Best Practices
● The Scheduler and Job Submission● Compute Canada and Calcul Québec
4
Introduction
● This workshop is built from the HPCS 2015 - Introduction to ARC workshop:○ Etherpad with extra information:
■ https://etherpad.mozilla.org/hpcs2015-intro-arc
● About Post-its○ Blue-green : OK, exercise done○ Orange-red : not OK, I have a question or a problem
● Presentation:○ Instructors and participants○ Name and research domain
6
Other research domains
● Genomics● Molecular Dynamics (MD)● Finite Element Analysis (FEA)● Computational Fluid Dynamics (CFD)● Astronomy and Astrophysics● Geospatial Data Analysis● 3D renderings, Image Analysis● Artificial Intelligence / Deep Learning (AI)● Humanities and Social Sciences (DH)
7
Advanced Research Computing
● High Performance Computing (HPC)○ Parallel code○ Profiling and optimizing code○ Running code on accelerators
● Management of massive data● Complete “pipelines” for jobs submission● Sharing data● Web portals for special projects● Cloud computing / storage
8
ARC is also About Having Enough Resources● https://www.socrative.com/
○ Student login - room PIERLUCSTONGE● Problem: we want to compute a dense matrix
multiplication C = A * B○ Each square matrix is made of ~1 billion floating
point numbers in double precision● How much memory do you need for this job?
a) 3.2 GBb) 4.3 GBc) 8.6 GBd) 12.9 GBe) 25.8 GB
9
Different Tools for Different NeedsPortable computer
● 2-4 cores● 4-16 GB● Local storage, USB
○ 128 GB to 4 TB● Wi-Fi and Ethernet
○ General usage○ Cloud storage
● Gaming GPU○ 32 bits
● Direct access
10
HPC Compute node
● 8-64 cores● 16-3072 GB● Network storage
○ 150 TB to 16 PB● Ethernet and fast
network (IB, OPA)○ Multi-node tasks
● Accelerators○ 64 bits, ECC mem.
● Access by queued jobs
Zoom on a HPC Compute Node
11
Local hard-drive (scratch)
Proc. Proc.
Proc.Proc.
RAM
RAM
RAM
RAM
RAM
RAM
RAM
RAM
Proc. Proc.
Proc.Proc.
Infiniband/Omni-Path
Ethernet
Accelerator
Why Infiniband and Omni-Path?
12
● Some tasks need more than one compute node○ Split workload, memory usage and storage access
● High bandwidth○ Ethernet: 1 Gbps (~100 MB/s)○ Infiniband and Omni-Path: 20-100 Gbps (2-12 GB/s)
■ Best for I/O on large files
● Reduce latency○ Ethernet: ~30µs (33k iops = I/O operations per sec.)○ IB and OPA : 2 µs or less (500k iops or more)
■ Best for several small communications
Storage Available on Compute Nodes
13
Project spaceScratch space
Bandwidth
Latency
Local scratch
RAMDISK
Capacity
(Not to scale)
Typical Cluster
14
Ethernet &
IB or OPA Login node
ssh me@...
InternetStorage
homeprojet
scratch
Scheduler Nodes
Job Job
Typical Cluster Usage
15
1. Connect to a login nodea. Import datab. Compile code or install applications
2. Submit jobs to the schedulera. Define needed resources: processors, accelerators,
memory, walltime limitb. Wait in queue - monitor your positionc. Automatic execution of your jobs on compute
nodes
3. Get your results
Typical Environment on Cluster● The operating system is usually Linux (CentOS,
Scientific Linux, …)● To connect to a cluster, we usually use an encrypted
session through SSH (Secure SHell)○ To transfer data safely: SCP (Secure CoPy) and
others tools…○ Some web portals may also provide an access
● Development environment○ Multiple text editors○ Scientific software○ Version control software
● Tools for transferring large amount of data○ 10 Gbps, 40 Gbps or 100 Gbps between sites
16
Moving Data Over SSH
● Secure CoPy with SCP or WinSCP:○ Most clusters only allow SSH connections
■ SSH: Secure SHell (username and password)○ SCP uses the SSH protocol to copy files
■ Simple tool for small amount of data (<1GB)○ On Windows: use WinSCP, FileZilla or MobaXterm
■ Two panels, where one side shows your files on the cluster, then drag-and-drop files or folders
○ On Linux or Mac OS X: use the scp command:■ scp file.txt user@cluster:folder/■ scp -pr user@cluster:folder ./
18
Globus - Why?
● Secure CoPy with SCP or WinSCP:○ One process and one thread○ Fails on dropped connections
● BBCP - parallel transfers○ Need a calibration for the round-trip time○ Command line with multiple arguments
● Globus - Web interface and CLI○ No need for calibration○ Convenient Web interface○ Confirmation email after the transfer○ Possible to share data with other users
19
Go to Compute Canada Research Portal● CC > Research Portal > National Services > Data Movement (Globus)
https://www.computecanada.ca/research-portal/globus-portal/
20
Create an account if you do not have
OrLog in to your account if you already have an account
Account Creation and Login1. Go to: https://globus.computecanada.ca/SignIn#step=signup2. Select Compute Canada if you have a CCDB account, or Globus ID if
not. To create a Globus ID account: https://www.globusid.org/create
21
Exercise: Sign-up!
23
● https://globus.computecanada.ca/SignIn#step=signup● Globus ID credentials are stored on Globus:
○ Can be a different username than the one on CCDB, or it can be the same. But both accounts (on CCDB and Globus ID) are different!
○ For sure, use a different password!
● Put the green Post-it on the computer screen when done
Globus Home Interface
24
To Logout
Globus Endpoint on your
computerTransfer options
Endpoint Viewports
Authentication to an Endpoint
26
● The authentication is done on a server corresponding to the endpoint○ Globus will never know your credentials for a given
cluster/endpoint○ The authentication server only grants access to
Globus.
Exercise: Import the Material
● Connect to computecanada#guillimin○ Use the class account credentials (on your paper)
■ class##■ **********
● Connect to intro-arc_guillimin_plstonge○ Select both exercises and pictures folders○ Click on the blue button to initiate the transfer to
your (class##) home directory
● Wait for the email report○ When the transfer is completed, put the green
Post-it on top of the computer screen28
Confirmation Email
29
TASK DETAILS
Task ID: 5ef1f2c6-81f5-11e6-afd7-22000b92c261
Task Type: TRANSFER
Status: SUCCEEDED
Source: intro-arc_guillimin_plstonge (...)
Destination: computecanada#guillimin (...)
Label: n/a
https://www.globus.org/app/activity/...
Create a Local Endpoint
33
1. Choose a unique Endpoint namea. Generate the Setup Keyb. Copy the Setup Key in a text editor
2. Download and Install Globus Connect Personala. Start it and paste the Setup Key
3. A Globus icon ( g) appears in the taskbara. [Right-]click -> Preferences -> Accessb. Select accessible directories you want to export
4. Use your Endpoint: user#uniqueName
Vocabulary
● Serial tasks○ Any task that cannot be split in two simultaneous
sequences of actions○ Examples: starting a process, reading a file, any
communication between two processes
● Parallel tasks○ Data parallelism: same action applied on different
data. Could be serial tasks done in parallel.○ Process parallelism: one action on one set of data.
Action split in multiple processes or threads.■ Data partitioning: rectangles or blocks
36
Parallel tasks
● Shared memory○ Multiple threads share the same memory space in a
single process: full read and write access.
● Distributed memory○ Each process has its own memory space○ Information is sent and received by messages
● Parallel efficiency (scaling)○ Amdahl’s law: how long does it take to compute a
task with an infinite number of processors?○ Gustafson's law: what size of problem can we
solve in a given time with N processors?
37
Connecting to Guillimin - SSH Client
● From Linux, open a Terminal window● From Mac OS X, open a Terminal window● From Windows, install PuTTY, MobaXterm
or any other SSH (2.0 compliant) client○ http://www.chiark.greenend.org.uk/~sgtatham/putty
/download.html■ putty.exe is a standalone executable■ Simply run putty.exe
○ http://mobaxterm.mobatek.net/
39
Connecting to Guillimin - Terminal
● Server name: guillimin.calculquebec.ca● Username: class##
○ ## = Two digits on the first line of the small piece of paper
● Password: **********○ Sequence of 10 characters on small piece of paper
● Linux/Mac OS X example:ssh [email protected]
○ Accept the server’s key fingerprint
40
Scientific Software
● You will use or create scientific software○ Applications○ Libraries
● Make sure you are using a code language widely used in your research domain
● Make sure your code is well documented and easy to maintain○ Keep in mind that future researchers are going to
use your software
● Do not reinvent the wheel: use modules!42
Modules
● Exercise:module avail
● Compilers: GNU, Intel, LLVM, PGI, Java● Language interpreters: Python, R● Libraries and other applications:
○ OpenMPI, MVAPICH2, Boost○ MKL, FFTW○ GROMACS, NAMD, Quantum-Espresso, NetCDF
● Other tools:○ Paraview, GNU-Parallel, GDB, Valgrind
43
Exercises with Moduleswhich gcc
module add gcc/4.8.2
module list
which gcc
module rm gcc/4.8.2
module add ifort_icc openmpi
echo $LD_LIBRARY_PATH
echo $CC
module purge
44
Lmod/EasyBuild-based hierarchical module system● A default set of modules on Cedar and Graham● Some legacy and EasyBuild modules are available
on Guillimin. To look for any R: module spider R
Exercise on Guillimin:1. Load the main toolchain Intel + OpenMPI + MKL:
module load iomkl/2015b
2. List loaded modules:module list
3. You now have access to much more modules:module avail
45
Exercises withCompute Canada Modulesbash
source /software/soft.computecanada.ca.sh
module list
module avail
module load boost
module purge
exit
46
Introduction to GNU-Parallel● https://wiki.calculquebec.ca/w/GNU_parallel/en● Automatic online process scheduler for serial tasks
○ User provides a command line template and one or multiple sequences of arguments
● For N cores on a compute node, GNU-Parallel will start N processes○ As soon as one process is complete, it will start the
next process according to values in sequences● Example:
module load parallel/20150822
parallel echo {1} ::: $(seq 1 960)
47
Introduction to Threads
● Threads are used for parallel tasks on a single node
● One process, one shared memory space, but multiple threads
● Synchronization issues, like race conditions○ Two threads reading or writing to the same variable
● Instead of POSIX threads:○ OpenMP○ Threading Building Blocks
48
Introduction to MPI
● Message Passing Interface (MPI) is a standard for passing messages between multiple processes on multiple nodes○ Each process can know its rank number
● Same set of functions, but different implementations○ MPICH2, MVAPICH2, OpenMPI, Intel MPI,...
● Example:module load iomkl/2015b
mpiexec -n 4 date +%s.%N
49
Version Control &Revision Control
● Bad practice:○ Keeping multiple versions in sub-directories○ Sending a specific version by email○ Trying to merge two versions manually
● Good practice: using a version control system○ Choose one that fits your project:
■ Git (git)■ Mercurial (hg)■ Subversion (svn)
○ Repository: shared on a remote server
51
Exercises with gitmkdir -p ~/projectA; cd ~/projectA
module add gitgit init # Local repos.
touch script.sh; git statusgit add script.sh # Ready to commit
git commit -m "intro-ARC" # Do commit
Add line “env > env.log” in script.shgit diff # See differencesgit add script.sh # Ready to commitgit commit -m "Variables" # Do commit
git log # Show past commits
52
One Last Good Practice -Using SSH Keys
● Having access with a local passphrase:ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (~/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
● Copy the public key to a remote site (id_rsa.pub):
scp .ssh/id_rsa.pub <user>@<server>:
ssh <user>@<server> \
"cat id_rsa.pub >> .ssh/authorized_keys"
53
Exercise - Code Compilation
54
● Go to the exercises folder
cd ~/exercises
● Load the following modules:
iomkl/2015b
Boost/1.59.0-Python-2.7.10
ImageMagick/7.0.1-6
● Build the application:
make
The Scheduler
● Jobs are not running on login nodes● You need to submit your jobs to the
scheduler:○ Resources you need: n processors, memory,
accelerators○ Job duration○ Account name○ The script to run on worker nodes○ Where you want your results
https://wiki.calculquebec.ca/w/Running_jobs
56
Schedulers in Calcul Québec
57
Job component Torque + Maui/Moab
Submission commandTorque: qsub
Maui/Moab: msub
Account -A xyz-123-aa
Processors (small L)Some clusters force full nodes
-l nodes=N:ppn=P-l procs=PP
Memory (small L)-l pmem=1700m-l mem=20g
Job duration (small L) -l walltime=d:hh:mm:ss
Job name -N name
Where you want your results-o output_path-e error_path
your_app > out.log
Environment Variables During Job
58
Variable Description
$PBS_JOBIDJob IDEx.: 12345678.server.name
$PBS_JOBNAMEJob name set by -NEx.: introARC
$PBS_NODEFILEThe name of the file that contains a list of assigned nodes for the job
$PBS_O_WORKDIRThe current working directory ($PWD) when the job has been submitted.
$PBS_ARRAYIDOne of the values specified by the job array: -t 1-3,6,10-12
Demo:Edit and Submit a Job Script
59
● Go to projectA: cd ~/projectA
● Edit with: nano script.sh#!/bin/bash#PBS -l nodes=1:ppn=1#PBS -l walltime=5:00#PBS -N myEnvenv > env.log
● Submit job: qsub script.sh
Demo:Manage your Jobs
● Torque - jobs and job array detailsqstat -u $USERqstat -t -u $USER
● Maui / Moab - jobsshowq -u $USER
● (Optional) Torque - cancel jobqdel jobid
● (Optional) Maui / Moab - cancel jobmjobctl -c jobidcanceljob jobid
60
Scheduler on Cedar and Graham
● Slurm Workload Manager○ sbatch script.sh
■ #SBATCH --account=def-someuser■ #SBATCH --time=3:00 ## 3 minutes■ #SBATCH --nodes=2■ #SBATCH --ntasks-per-node=8■ #SBATCH --cpus-per-task=4■ #SBATCH --mem-per-cpu=3G
○ squeue -u $USER○ scontrol show job -dd JOBID○ scancel JOBID
61
Exercises:Edit and Submit a Job Scripts
● Go to the ~/exercises folder○ Check the README.en file for general instructions
● Start with the exercise 1-base. All the instructions are in the README.en file○ The solution is in solution.sh
● Then try exercises 2, 3 and/or 6○ For exercise 2, a description of “&” and “wait” is in
the README.en file of the exercise○ You need to understand the solution of the exercise
3 to do exercise 4
62
Compute Canada
● Four consortia (from West to East):○ WestGrid
■ British Columbia, Alberta, Saskatchewan, Manitoba
○ Compute Ontario (CAC + SciNet + SHARCNET)■ Ontario
○ Calcul Québec■ Québec
○ ACENET■ New Brunswick, Newfoundland, Nova Scotia,
Prince Edward Island
64
Example - Calcul Québec Consortium
65
● Compute Canada○ Calcul Québec - Compute Resources
■ Briarée & Hadès (U. Montréal)■ Colosse & Helios (U. Laval)■ Guillimin (McGill)■ MS2 & MP2 (U. Sherbrooke)
○ Calcul Québec - Human Resources■ About 40 employees
● ARC analysts● System administrators● Scientific and operational directors
Biggest Clusters
66
Name / Institution Cores GPUs Summary
Niagara / U. Toronto ~60000 0 Specifications to be confirmed
Graham / U. Waterloo 33472 320 32 cores/node, 128+ GB/node, E/FDR IB
MP2 / U. Sherbrooke 30984 0 24 cores/node, 32+ GB/node, QDR IB
Cedar / SFU 27696 584 32 cores/node, 128+ GB/node, OPA 100
Guillimin / McGill U. 17240 112 12,16 cores/node, 24+ GB/node, QDR IB
Orcinus / U. British Columbia
9616 0 8,12 cores/node, 16+ GB/node, D/QDR IB
Orca / SHARCNET 8320 0 16,24 cores/node, 32 GB/node, QDR IB
Briarée / U. Montréal 7560 0 12 cores/node, 24+ GB/node, QDR IB
Example of job requirements: 8, 16 or 32 cores/node, 30 GB/node, QDR IB
Roles and Allocations
67
Prof
CCRI : abc-123-01 ABCD UniversityAlloc.: abc-123-aa (default alloc., Rapid Access Service or RAS)
CCRI : abc-123-02 DEFG UniversityAlloc.: abc-123-ac 200 core*years
Stu-dent
Prof
CCRI : def-456-01 HIJK UniversityAlloc.: def-456-ab 50 core*years
xyz-789-02
xyz-789-01
About Allocations
68
● Amount of compute cycles (only a target):○ 1 core provides 2 to 3 GHz (cycles per second)○ Core*seconds, core*hours or core*years (cy)○ Each CQ site has a default allocation (up to 80 cy)
● Amount of storage space (for CQ):○ Private scratch space (default: up to 1TB)○ Group shared project space (default: up to 1TB)
● Special allocations (RRG and RPP)○ Resource Allocation Competition (RAC 2018)https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions/
How to Access CQ Resources - The Complete Procedure
● CCDB: https://ccdb.computecanada.ca● A professor creates an account in CCDB
○ Request role of Principal Investigator (PI) - CCRI○ (Optional) Request a special allocation (RAC)
■ Special allocation granted on a specific cluster● A student creates an account in CCDB
○ Request sponsored role - need to provide the PI’s CCRI
○ Apply for a consortium account (ex.: Calcul Québec)○ In the Calcul Québec portal, activate your access to
any cluster■ Choose clusters based on special allocations
69
Usage Policy
● Each user has his own account; never share your private account!
● No heavy process (in time and space) on login nodes
● Resources must be used in an optimal way○ Large parallel jobs must use tools that scale well○ “Cold” data should not stay on disk○ Maximize the amount of research done in Canada
● Must acknowledge Compute Canada and any regional partner organization
70
Service and Support● Training workshops:
○ Future: http://calculquebec.eventbrite.ca○ Past:
■ https://wiki.calculquebec.ca/w/Formations/en■ http://www.hpc.mcgill.ca/index.php/training
● Documentation:○ https://www.computecanada.ca/research-portal/○ https://docs.computecanada.ca/wiki/Compute_Canada_
Documentation○ https://wiki.calculquebec.ca/w/Accueil○ http://www.hpc.mcgill.ca/index.php/starthere
● Support:○ [email protected]
71
Conclusion
● Need help?○ WestGrid: [email protected]○ SciNet: [email protected]○ SHARCNET: https://www.sharcnet.ca/my/security/login○ Calcul Québec: [email protected]○ ACENET: [email protected]
● Useful information to send us:○ User name○ Job ID (if applicable)○ Path to {job script, output/error files, code} or as
attached files
73