Introduction to Linux

September 24, 2015

1

By: Simon Nderitu

Partners and Sponsors

2

3

Schedule and Outline● 9:00-10:00 The Terminal● 10:00-11:30 File Systems

○ 10:20-10:30 Break● 11:30-13:30 Running Commands

○ 12:00-13:00 Lunch● 13:30-14:30 Processes

○ 14:20-14:30 Break● 14:30-15:00 How to Compile● 15:00-15:30 Remote Connections● 15:30-16:00 Graphical User Interfaces in Linux

Short History of Linux● Reference:

○ http://www.cs.cmu.edu/~awb/linux.history.html● Project : started in 1991 by Linus Torvalds● Context :

○ Unix vs DOS○ New Intel 386 architecture

● Linux kernel:○ Open source○ Under the GNU GPL

● Linux used in high-performance computing clusters

4

The Terminal

5

The Terminal● How to Open a Terminal● Bash● How to Exit● Previous Commands in Bash● Auto-completion in Bash● User Manuals● Help Topics● Environment Variables (1-2)● Exercises

6

How to Open a Terminal● How to log in a local computer

○ Username: csuser##○ Password: ___@[S##

■ Replace ## with both digits on the first line of the small piece of paper

■ “___” are 3 secret letters provided in classroom● In the Ubuntu interface, open a Terminal

○ On top left of the screen, click on the Ubuntu button○ Type: terminal○ Select the Terminal application

7

Bash● Bash is a shell:

○ Gives access to file systems○ Runs commands as a specific user○ Has per-session environment variables

● Started from a terminal, a parent shell or SSH.○ In Ubuntu, search for application “Terminal”.

● The Bourne-Again shell (Bash) is the default shell on Guillimin and on many other linux systems.

● Other shells : Bourne shell (sh), C shell (csh), Tenex C shell (tcsh), Korn shell (ksh), etc.

8

How to Exit● Type “exit” (w/o quote marks) and press Enter

○ The “exit” command will go to the history● Or use Ctrl+D

○ Bash receives commands from the STDIN stream○ Ctrl+D closes the STDIN stream○ Bash will exit itself as soon as STDIN is closed○ This exit will not be logged in the history

● If Bash was started by:○ A terminal : the terminal will close as well○ A parent shell : the user will return to the parent shell○ SSH : the connection will be closed

9

Previous Commands in Bash● To get all the logged previous commands, type:

history

● To get a previously executed command, use the Up arrow key until you find it

● You may search for a command in the history:○ Use Ctrl+R○ Start typing parts of the command you are looking

for○ Use the Left or Right arrow keys to select and edit

the command (if needed)○ Use Ctrl+C to cancel

10

Auto-completion in Bash● The first word in a command is usually an

executable:<executable>

● The other words (or arguments) may be options or short or long paths to files or folders:

<executable> -a -b --option --option2<executable> -ab -- <filename>

● Knowing that, Bash can auto-complete what you are typing or give you some suggestions

● Use the Tab key to trigger the auto-completion function

11

User Manuals● Some programs have one or many manual pages

man <program-name>

● Most of the man pages are also on the Web● In the man page, you can search for a keyword:

○ Type “/”○ Type your keyword and press Enter○ All instances of the keyword will be highlighted. Man

will go to the first instance. Press the “n” key to go to the next one.

● To quit the man page, press the “q” key○ Man pages are displayed (piped) through a program

called 'less'

12

Help Topics● Some programs provide additional help topics on

the command line:

executable --help

executable -h

executable help

13

Environment Variables● To print a list of the current environment

variables:

env

● Notable variables:○ HOSTNAME: computer's name○ PWD: current directory○ SHELL: default shell (Bash)○ LANG: session language, text encoding

14

Environment Variables (2)● Other notable variables:

○ LOGNAME, USER: session user name○ GROUP_ID: groups of current user○ HOME: home directory○ PATH: paths to executable files○ LIBRARY_PATH: paths to libraries (mainly for

compilation)○ LD_LIBRARY_PATH: paths to libraries (mainly for

execution)

15

Exercises● Use the auto-completion to reach the following

executable and print the help topics:

/usr/bin/svn

○ /u<Tab>b<Tab>...● Connect to Guillimin and get the hostname:

ssh guillimin.hpc.mcgill.ca -l class##

env

● What is Python? Read its manual page:

man python

16

File Systems

17

File Systems● Where am I?● Structure of Directories● Mounted File Systems● Files and File Attributes (1-4)● Changing File Attributes● Directories and Directory Attributes (1-2)● Changing Directory Attributes● Symbolic and Hard Links● Exercises (1-2)

18

Where am I?● Print working directory:

pwd

● List the files and folders in the current directory (# for comments):

ls # Same as ls * where * could be anything

ls -a

● Hidden files and folders start with a dot “.”● The “.” folder is the current directory● The “..” folder is the parent directory● To change the directory (use “;” between commands on

one line):cd <directory_name_or_full_path>

cd .. ; cd - # parent, previous

19

Structure of Directories● / : root directory of everything in Linux● /home : contains all home directories● /tmp : temporary files and folders● /bin, /usr/bin, /usr/local/bin, etc. :

○ see the PATH variable (directories for executable files)

● /dev : devices and virtual devices○ /dev/null, /dev/random, /dev/urandom, /dev/zero

● /proc : processes information○ /proc/meminfo, /proc/cpuinfo

20

Mounted File Systems● Directories are mount points

/media/USB-DRIVE

● Print the mounted file systems:mount

● Usage of the mounted file systems:df -h

● Your personal quota (if any):quota # or myquota on Guillimin

● Disk usage of a file or a directory:du -sh <file_or_directory>

21

Files and File Attributes● Create a file or change its time stamps:

touch <filename>

● Show the attributes of a file:stat <filename>

ls -l <filename>

● Rename or copy a filemv <filename> <newname> # No confirmation

cp -a <filename> <newname> # Preserves all

● Remove a file permanently (no trash/recycle bin):rm <filename> # No confirmation

22

Files and File Attributes (2)● Current language (LANG):

locale

● Edit a new or an existing text file (nano, emacs, vi):

nano -w <filename>

● Keyboard shortcuts in nano:○ Ctrl+X : Exit○ Ctrl+O : Save○ Ctrl+K : Cut the current line. Repeat for consecutive

lines○ Ctrl+U : Paste the multiple lines

23

Files and File Attributes (3)● Get the type of file (text encoding, image format, ...):

file <filename>

● View a text file:less <filename>

■ Press “q” to quit

more <filename>cat <filename>

● Word count and line count:wc <filename> # All statisticswc -w <filename> # Word count onlywc -l <filename> # Line count only

24

Files and File Attributes (4)● See the default file attributes:

ls -l # -rw-rw-r-- username group

umask # 0002

id # uid=... gid=... groups=...

● Permissions:○ - : regular file○ rw- : user has read and write access, no exec.○ rw- : group has read and write access, no exec.○ r-- : other users have read access only

25

Changing File Attributes● Changing the permissions:

chmod u+x,o-r <filename>

chmod 760 <filename> # -rwxrw----

● 760 is an octal number : 421 42- ---chmod 640 <filename> # -rw-r-----

● Changing the ownership:chown <username> <filename>

chown <username>:<group> <filename>

chgrp <group> <filename>

26

Directories and Directory Attributes● Go to your home directory and list its attributes

cd ~

cd

ls -ld . # drwx------ username group

ls -la

● To access inside a directory, the execute “x” permission must be enable.

● Hidden directories start with a “.”

27

Directories and Dir. Attributes (2)● Create a directory:

mkdir <dirname> # mkdir -p very/long/new/path

● Show the attributes of the directory:stat <dirname> ; ls -ld <dirname>

● Rename or copy a directorymv <dirname> <newname>

cp -r <dirname> <newname> # cp -pr OR cp -a

● Move a file or a directory into another directory:mv <file_or_dir> <dirname>/[<newname>]

● Remove a directory (no trash/recycle bin)rmdir <emptydir>

rm -r <dirname>28

Changing Directory Attributes● Remove other accesses recursively:

chmod -R o-rwx <dirname>

● Add the group sticky bit recursively:chmod -R g+w <dirname>

find <dirname> -type d -exec chmod g+s {} \;

● Changing the ownership recursively:chown -R <username> <dirname>

chown -R <username>:<group> <dirname>

chgrp -R <group> <dirname>

29

Symbolic and hard links● Create a symbolic link (files and directories):

ln -s <original_file_or_dir> <linkname>

ls -l

● Create a hard link (files only):ln <original_file> <linkname>

ls -li # inode_ID ... num_links

● Remove a link:rm <linkname>

30

Exercises● Create a text file. Write something in it.● Create a directory. Move the file into the directory.● Block all other users access (rwx) to this directory.● Create a second directory. Move the first directory

in this second directory.● Create a symbolic link to the first directory.● Create a hard link to the text file. Modify the file

from the new hard link.● Print the content of “both” files.● Remove everything created in these exercises.

31

Exercises (2)● Import a compressed archive from Internet:

mkdir www; cd www

wget http://www.hpc.mcgill.ca/downloads/

intro_linux/intro-linux201509.tar.gz

● Extract the content:tar -xvf intro-linux201509.tar.gz

● Create another tar archive:tar -cvf archive.tar intro-linux

● Compress the new archive:gzip archive.tar

32

Running Commands

33

Running commands● Which, Where?● What Time?● Hello World!● Filtering (1-2)● Bash Scripts● The .bashrc file● Exercises

34

Which, Where?● Which command is this?

which <executable>

● Where is my file? Where is this program?locate <file>

● Find a file in a directory:find <dirname> -name <file_or_dir>

35

What Time?● The current date and time:

date

● Time to go to sleep?time -p sleep 123.45

● Want to stop a command running?Ctrl+C

36

Hello World!● Say hello and world (two arguments):

echo "Hello" $(echo -n "World")

● Pipe between output and input of two processes:echo "Hello World" | less

● Output redirection to a file (if the file exists, its previous content will be lost):

echo "Hello World" > file.txt

● Output redirection appended to the end of a file:date >> file.txt

● Input from a file:less < file.txt

37

Filtering● Basic regular expressions / searching in files:

cat file.txt | grep -i hello # Case insensitive

grep -n -i hello file.txt # Show line number

grep some_string file1 file2 file3

● Extended regular expressions:grep -E "^k?[hH]el+o.{2}o.*$" file.txt

○ ^: beginning of line, $: end of line○ ?: 0 or 1 occurrence, *: 0 or many occurrences○ +: 1 or many occurrences, {n}: exactly n occurrences○ .: any character○ [a-zA-Z0-9]: list of characters (matches only one)

38

Filtering (2)● A .csv file (intro-linux/examples/):

Animal,Color,Age

cat,black,10

dog,black,5

dog,brown,8

● Show all dogs:awk -F, '/dog/ {print $3,$2}' file.csv

grep dog file.csv | awk -F, '{print $3,$2}'

39

Bash Scripts● Consider script.sh:

#!/bin/bashecho "------------"

date

ls -a

● Make script.sh executable:chmod +x script.sh

● Run script.sh:./script.sh > myfiles.txt

40

The .bashrc file● Located in your home directory:

cat ~/.bashrc

● This is a Bash script that is executed at the beginning of each Bash session

● Used to set environment variables and aliases● Example of an alias:

alias rm='rm -i' # Adds confirmation

41

Exercises● Use the .bashrc file, echo, hostname, date

and output redirection to log all new Bash sessions in ~/.mysessions with the following format:

New session on <hostname>: <date>

● Open and close Bash a few times.● Print only the date of the last 4 sessions (use

tail -n, the separator would be “: “ for awk).

42

Processes

43

Processes● Running Processes● How to Kill Processes● Processes in Background● Introduction to Threads● Exercises

44

Running Processes● Listing the running processes in your session:

ps

● Listing all the running processes on the server:ps -ef

● The init process (PID = 1)● The parent PID (PPID)● Monitoring the running processes (“q” to quit):

top

45

How to Kill Processes● Sending signals to a process:

kill -<number> PIDkill -s SIGNAL_NAME PID

● To list the different signals:kill -l

● The default termination signal is SIGTERM (15):kill PID

● The termination signal from a terminal (like Ctrl+C):kill -s SIGINT PID # or kill -2 PID

● To force the process to stop now:kill -s SIGKILL PID # or kill -9 PID

46

Processes in Background● Processes unattached to a terminal : they are running

in background● To start a process directly in background:

command arg1 ... > outputfile &jobs

● To resume in foreground:fg

● To suspend a foreground process (SIGSTOP): Ctrl+Z● To resume in background (SIGCONT):

bg

● To wait for all background processes to terminate:wait

47

Introduction to Threads● One or many threads in a process

○ CPU usage may be greater than 100%● Shared memory

○ False sharing● Synchronization concepts:

○ Barrier: synchronisation○ Mutex: one and only one thread○ Semaphores: wait and release

48

Exercises● Start a secondary bash session in your primary

session● Start the following command

dd if=/dev/urandom of=/dev/null

● Put it in the background○ Suspend the command○ Resume in the background

● While running, look at the CPU usage● Quit the secondary bash session● Get the new PPID of dd● Kill the dd process with SIGTERM

49

How to Compile

50

How to Compile● C with OpenMP Code● C with MPI Code● C++ Code● Makefiles● Perl Scripts

51

C with OpenMP Code● Code example

main.c

● Compilationgcc -o exec_ser main.c

gcc -o exec_omp main.c -fopenmp

● Executiontime -p ./exec_ser

time -p ./exec_omp

52

C with MPI Code● Code example

main.mpi.c

● Compilationmodule add ifort_icc # On Guillimin

module add openmpi

mpicc -o exec_mpi main.mpi.c

● Executiontime -p mpiexec -n 2 ./exec_mpi

53

C++ Code● Code example

main.c

● Compilation

g++ -o exec_cpp main.c

● Execution

time -p ./exec_cpp

54

Makefiles● A Makefile file

○ Variables (set: VAR=..., use: $(VAR))target: dependencies commands ...

○ Regular expressions%.o: %.cpp

○ Special variables ($@, $<, $+)● The make command

make -f <makefile>make # if Makefile or makefilemake <target>

55

Perl Script● Code example

main.pl (-rwxr-x---)

● Code validationperl -c main.pl

● Execution

perl main.pl

./main.pl

56

Remote Connections

57

Remote Connections● Using SSH● Using SCP● Using SSH Keys

58

Using SSH● From a terminal (Mac OS X, Linux):

ssh <username>@<server.name.ca>

ssh <server.name.ca> -l <username>

○ (Accept the certificate) Enter the password● Connect to a different port number:

ssh <username>@<server.name.ca> -p <num>

● From PuTTy (Windows, Linux):○ Host Name: <server.name.ca>○ Click on “Open” (Accept the certificate)○ Login as: <username>, Enter the password

59

Using SCP● Secure copy with SSH protocol● From a terminal (Mac OS X, Linux)

scp filename <user>@<server>:/full/path/

scp filename <user>@<server>:directory/

scp filename <user>@<server>:newname

scp -pr dirname <user>@<server>:

scp * <user>@<server>:directory/

● From WinSCP (Windows)○ Server name, credentials, drag & drop○ Beware of the text copy mode (\n vs \r\n)

60

Using SSH Keys● Having access with or without a passphrase:

ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (~/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

● Copy the public key to a remote site (id_rsa.pub):

scp .ssh/id_rsa.pub <user>@<server>:

ssh <user>@<server> \

"cat id_rsa.pub >> .ssh/authorized_keys"

61

Graphical User Interfaces in Linux

62

Graphical User Interfaces in Linux● Desktop Environments and Shell Interfaces● The X Server

63

Desktop Environments andShell Interfaces● Gnome

○ http://www.gnome.org/gnome-3/○ https://unity.ubuntu.com/about/

● KDE○ http://userbase.kde.org/An_introduction_to_KDE

● Linux Mint Cinnamon○ http://cinnamon.linuxmint.com/

● LXDE○ http://lxde.org/image_galleries/screenshots

● MATE (Gnome 2 fork)○ http://mate-desktop.org/index.html

● Xfce○ http://www.xfce.org/about/screenshots

64

The X Server● Decoupling the application and the rendering

engine○ X Client: OS, application, remote server○ X Server: keyboard, mouse, screen

● Using SSH: -X or -Yssh -X <user>@<server.name.ca>

● X Server for Windows : Xming, MobaXtermhttp://sourceforge.net/projects/xming/http://mobaxterm.mobatek.net/

● Open remote applicationcommand args... &

65

Conclusion

66

Conclusion● Try Linux in a virtual machine (Virtual Box)● Many distributions

○ http://en.wikipedia.org/wiki/List_of_Linux_distributions

○ Arch Linux, CentOS, Debian, Fedora, Gentoo, Linux Mint, Ubuntu, etc.

● Web pages are friendlier than Man pages

● Questions?guillimin@calculquebec.casupport@calculquebec.ca

67