Introduction to Unix

2008 April

http://www.accre.vanderbilt.edu

Introduction to Unix 2

Outline

Overview of the Unix Operating System (slides 3-5)

Shell, CL, Basic Ops, Help, Special Characters (slides 6-16)

Files, Directories, & Access Rights (slides 17-18)

Defining Your Environment (slides 19-25)

Text Editors (slide 26)

Input/Output Redirection - Pipelines & Filters (slides 27-39)

Processes & Multitasking (slides 40-41)

Scripting (slides 42-45)

Compressing & Archiving (slides 46-47)

Getting Help from ACCRE (slide 48) ; O’Reilly books (slide 49)

screen, a nice Unix tool (slide 50)

Introduction to Unix 3

What is Unix?

Unix is an operating system (OS) like Windows

UNIX was originally created in the late 1960's by Ken Thompson and Dennis Ritchie of AT&T Bell Labs

It was created out of frustration with MULTICS (UNICS was a pun on MULTICS)

It was designed to be a programmers OS

It turned out to be a portable, multitasking, multiuser OS

Introduction to Unix 4

Unix or UNIX?

UNIX is trademarked by the Open Group

To be called UNIX, an operating system must be certified as meeting the specification (currently UNIX 03) by the Open Group

Four UNIX's: Solaris (Sun), AIX (IBM), HP-UX (Hewlett-Packard), MacOS X Leopard (Apple)

Unix refers to any of the “Unix-like” operating systems such as Linux, [Open|Free|Net]BSD, or pre-Leopard MacOS X

Introduction to Unix 5

All Unix's aren't the same

Each Unix usually develops its own niche or specialization

AIX / HP-UX – heavy duty business server

Solaris – e-mail / web server

MacOS X - “User friendly” desktop

Linux – High performance OS and free, which makes it ideal for HPC clusters

OpenBSD – ultra-secure server

Introduction to Unix 6

Unix Shell / Command Line The shell is an interpreter used to communicate with

the OS interactively on the command line (CL)

The cluster has two “flavors”: tcsh and bash. Online manuals and FAQ:

http://www.gnu.org/software/bash/

http://www.tcsh.org/Home

For historical reasons, tcsh was the default shell for all users on VAMPIRE (the ACCRE cluster); now it's bash

If you wish to change your default shell, use the chsh command on vmpsched

CL prompt> chsh -s /bin/bash

Introduction to Unix 7

CL Operations and Files

There are Unix commands (same function in any shell) and shell built-in commands

Commands entered at the CL prompt have options and arguments

Files and directories are the primary abstraction in Unix (similar to files and folders in Windows)

Directories are files with information on their contents

Some useful Unix commands for interacting with files and directories:

Introduction to Unix 8

Basic Commands

• pwd - print working directory• ls - list contents of directories• mkdir - make (create) new directories• cd - change the current directory• cp - copy files or directories• mv - move files or directories• rm - remove (delete) files or directories• cat - concatenate file contents• more/less - scroll file contents• file - show file type

simple bash examples with CL editing

Introduction to Unix 9

Command Help

Unix traditionally includes instructions and help files (manual pages) on most commands and API's and their options. To access a manpage:

CL prompt> man foo

For csh/tcsh and sh/bash built-in commands:

CL prompt> man builtin

Some commands have command line help (usage hints or --help)

Other sources of help: /usr/share/doc, Google

Introduction to Unix 10

Basic CL & Editing• <tab> complete filenames and commands on CL• up/down arrows scroll CL history• left/right scrolls back and forth on CL• history shell command lists CL history (also tcsh)

functionCL (event) repeat

(function described on next slide):

repeat most recent, but substitute new for first old^old^new^

repeat nth event (see history)!n

repeat nth previous event!-n

repeat last containing string!?string

repeat last starting with string!string

repeat previous command!!

Introduction to Unix 11

Basic CL & Editing

• “:” allows more complex CL editing, e. g., select last event containing specific word, replace word, then execute command:

CL prompt> !?old:s/old/new/

This repeats the last event containing old, but substitutes new for old

Introduction to Unix 12

Basic CL & Editing

temporary interrupt (can also send to background, slide 38)^Z

cancel current CL, return prompt^C

delete under cursor^Ddelete to end of line^K

back 1 word<esc>Bback 1 character^Bforward 1 word<esc>Fforward 1 character^Fend of line^Ebeginning of line^Anext line^Nprevious line^P

function on command linekeystroke

Introduction to Unix 13

Files, Directories, Special Characters

Files to care about• hidden (filenames begin with a dot, list with ls -a),

e. g., user initialization, .login, .bashrc, .cshrc, …• global initialization, e. g., /etc/bashrc• devices• symbolic links

/ = top root directory A few special Unix shortcuts for file/directory names/paths:

• ~ = expands word to your home directory path• ~username = home directory of any username• . = current directory• .. = parent directory

Introduction to Unix 14

Files, Directories, Special Characters

Blank space

Shell executes command in background (slide 31)&

Quote following special character as normal string (e. g., filnames containing spaces, or slide 17)\

Examples throughout presentation (esp. scripts)“ ` { } #Separates commands (see slide 15);

more special charactersUsed within [] denotes range of characters-Enclose set of characters, match any one by position [ ]Match any single character in filename?Match any character in filename*

wildcards

N. b., best to avoid using these characters in filenames

Introduction to Unix 15

Files, Directories, Special Characters

E. g., globbing (expanding wildcard to match pattern):

CL prompt> ls -1 *file1file2~file3

CL prompt> ls ?ile1file1

CL prompt> ls file[1-2]file1

CL prompt> ls file[a-z0-9]~file2~

Introduction to Unix 16

Multiple commands and ;

You can use the “;” operator to append many commands on one line:

CL prompt> date ; uptime

Tue May 8 11:41:26 CDT 2007

11:41 up 8 days, 22:26, 1 user, load averages: 0.81 0.69 0.67

Introduction to Unix 17

File Permissions

Files/directories have an owner and a group

The owner can grant read/write/execute permissions to three groups (the user, the user's group, and all others on the system)

prompt> ls -l /home/usernametotal 16drwxr-xr-x 2 username group 8192 May 1 15:59 certs/drwxr-xr-x 3 username group 8192 Sep 15 17:28 classes/

Introduction to Unix 18

File Permissions

These permissions are modified by the chmod command, e. g. :

CL prompt> chmod g+rx file

permits other users in the owner’s group to read and execute file

Introduction to Unix 19

Startup Files, Variables, Aliases

Two types, environment and shell, can be (re)initialized at any time.

If defined in .login, environment variables are set upon login

If defined in shell run command files (e. g., .bashrc, .cshrc, .tcshrc), shell variables set upon each instance of shell

Frequently used variables should be set in these files

Introduction to Unix 20

Startup Files, Variables, Aliases

default when spawned by other programsEDITOR

login nameUSERterminal typeTERMdefault printerPRINTERdirectories containing commandsPATHoperating systemOSTYPEname of computerHOSThome directory pathHOME

screen for X windows displayDISPLAYmachine architectureARCH

functionname

A few common variables common to both shells:

Introduction to Unix 21

Startup Files, Variables, Aliases

Important environment variables:

• PATH - points to executables

• LD_LIBRARY_PATH – points to libraries

Reference a variable with $ or ${}

echo and printenv commands:

CL prompt> echo $PATHCL prompt> echo ${PATH}CL prompt> printenv PATH

Introduction to Unix 22

Setting Variables

To assign a value to a variable (shells have different syntax):

Example of adding to your PATH:

rehash lets shell know PATH was just updated

set FOO = “bar”

setenv FOO “bar”tcsh

export FOO=bar

FOO=bar ; export FOObash

set path = ( $path $home/bin:/directory2 )

setenv PATH “${PATH}:${HOME}/bin:/directory2”tcsh

export PATH=$PATH:$HOME/bin:/directory2 bash

Introduction to Unix 23

Setting Variables

Example of setting a new variable to the output value of a command:

CL prompt> current_date_time=`date`CL prompt> echo $current_date_timeMon Apr 23 14:15:35 CDT 2007

Another example (note pwd vs. $PWD):CL prompt> echo "I am in `pwd` on $HOST”

I am in /home/username on vmps08CL prompt> echo "I am in $PWD on $HOST”I am in /home/username on vmps08

Introduction to Unix 24

Setting Command Aliases

Shorten frequently used or lengthy commands:• tsch syntax:

alias rm 'rm -i’

• bash syntax:alias rm='rm -i’

• other useful aliases:alias cp='cp -i'alias mv='mv -i'alias ls='ls --color-tty -F'alias ll='ls -laF'

Introduction to Unix 25

Setting Command Aliases

Return to default:

• “backslash” temporarily returns to default:

\rm junk

• unalias returns default for rest of session:unalias rm

rm junk

Introduction to Unix 26

Text Editors on Cluster

Fairly basic:

Nano (http://www.nano-editor.org/)

More advanced:

Vi / Vim (http://www.vim.org/)

Emacs (http://www.gnu.org/software/emacs/)

Create/edit a new file example.txt

Introduction to Unix 27

Input/Output Redirection

Most programs have three I/O streams:

• stdin – standard input

• stdout – standard output

• stderr – standard error.

They all default to the console ("console" means the keyboard for the input and the screen for the output)

Introduction to Unix 28

Input/Output Redirection

To redirect stdout of a program to a file:bash: myprogram 1> output.logtcsh: myprogram > output.log

To redirect stderr of a program to a file: bash: myprogram 2> error.log

To redirect both stdout and stderr to same file (order matters): bash: myprogram > combined.log 2>&1

To redirect both stdout and stderr separately:myprogram >output.log) 2> error.log

Introduction to Unix 29

Input/Output Redirection

( comm > ofile ) >& efilestdout/err to 2 filescomm < ifile > ofile 2> efilestderr to third filecomm < ifile > filecomm < ifile > fileredirect stdin/outcomm <<ccomm <<cstdin until “c”comm >> ofile 2>&1comm >>& ofilestdout/err to endcomm 2>> ofilestderr to end of filecomm >> ofilecomm >> ofilestdout to end of filecomm < ifilecomm < ifilestdin from file

comm >ofile 2>&1comm &> ofile

comm >& ofilestdout/err to file

comm 2> filestderr to filecomm > filecomm >ofilestdout to file

bashtcshfunction

Introduction to Unix 30

Input/Output Redirection

Example of “>>” operator to append information to a file:

prompt> date > foo

prompt> cat fooWed Aug 31 17:27:52 CDT 2005

prompt> date >> foo

prompt> cat fooWed Aug 31 17:27:52 CDT 2005Wed Aug 31 17:27:56 CDT 2005

Introduction to Unix 31

Input/Output Redirection

Use of single back quotes to substitute in the value of another command:

Compare to slides 14 and 22:

prompt> ls `find . -name “file[a-z0-9]~”`file2~

prompt> echo `find . -name “file[a-z0-9]~”`file2~

Introduction to Unix 32

Input/Output Redirection

script to redirect CL stdin/out to record session to a file (however, formatting not so nice)

To redirect output to “nowhere” use the null device, /dev/null

To redirect stdout to null device (tcsh/bash):myprogram >/dev/null

To redirect stdout and stderr to null (bash):myprogram >/dev/null 2>&1

Introduction to Unix 33

I/O Redirection - Pipes/Filters

Pipelines are a set of processes chained by their standard streams, so that the stdout of each process feeds directly as the stdin of the next.

Pipelines are defined using the “|” character.

E. g., use a pipe and tee to direct output of echo to both stdout and to a file:

prompt> echo "Hello World" | tee output.txt

comm 2>&1 | comm2comm |& comm2pipe stdout/err to comm2

comm | comm2comm | comm2pipe stdout to comm2

bashtcshfunction

Introduction to Unix 34

I/O Redirection - Pipes/Filters

Commands can be used as filters which take the output of a program and modify it. E. g., use a pipe to count words from the output of echo:

prompt> echo "Hello World" | wc -w2

Very useful filters include:• grep - Pattern matching• sed - Search and Replace• cut - Print specific columns• sort - Sort alphabetically / numerically• uniq - Remove duplicate lines from a file

Introduction to Unix 35

I/O Redirection - Pipes/Filters

grep example:

prompt> cat example.txtHello WorldGoodbye World

prompt> cat example.txt | grep HelloHello World

prompt> cat example.txt | grep -v HelloGoodbye World

Introduction to Unix 36

I/O Redirection - Pipes/Filters

sed example:

prompt> cat example.txtHello WorldGoodbye World

prompt> cat example.txt | sed “s/Hello/Goodbye/g”Goodbye WorldGoodbye World

Introduction to Unix 37

I/O Redirection - Pipes/Filters

cut example:

prompt> cat example.txt1,Hello,World2,Goodbye,World

prompt> cat example.txt | cut -d "," -f 2-Hello,WorldGoodbye,World

Introduction to Unix 38

I/O Redirection - Pipes/Filters

sort example:

prompt> cat example.txt2 Goodbye1 Hello2 Goodbye

prompt> cat example.txt | sort -n1 Hello2 Goodbye2 Goodbye

Introduction to Unix 39

I/O Redirection - Pipes/Filters

uniq example:

prompt> cat example.txt2 Goodbye1 Hello2 Goodbye

prompt> cat example.txt | sort -n | uniq1 Hello2 Goodbye

Introduction to Unix 40

Processes and Multitasking

To run a program in the background, use the “&” character (or “^Z” followed by “bg”):

prompt> myprogram &[1] 7895

myprogram is now running in the background as process id (PID) 7895

Whenever your process finishes, it will print “Done” to the console.

Introduction to Unix 41

Processes and Multitasking

To check on the status of your jobs running on the system, use the ps command

prompt> ps -a PID TTY TIME CMD 8095 pts/3 00:00:00 ps

You can get an expanded list by typingps agux, or by using the top command

Use uptime to check the load average (how hard system is working) on slowly responding machines

Introduction to Unix 42

Simple Shell Scripting

String Unix commands into a shell script#/bin/csh N. b., not a comment on line 1

To execute:tsch: prompt> source myscript01.csh

bash: prompt> . ./myscript01.sh

Or run as executable: prompt> chmod u+x myscript01.csh

prompt> myscript01.csh

prompt> ./myscript01.csh (if not in path)

Introduction to Unix 43

Simple Shell Scripting

Simple bash programming: myscript02.sh#!/bin/bash# define variablelist="Mercury Venus Earth Mars Jupiter Saturn Uranus

Neptune”# initiate counteri=1# for loopfor planet in $listdo echo "planet $i is $planet" # Print to STDOUT i=`expr $i + 1` # Increment counterdone

Execute it:prompt> chmod u+x myscript02.sh ; ./myscript02.sh

Introduction to Unix 44

More on Variables

Can also define conditional variables:

If VAR, use its value else use alternate_value and exit if in shell script; if alternate_value also empty print print msg to stderr

${VAR:?alternate_value}

If VAR, use alternate_value${VAR:+alternate_value}

If VAR, use its value else use alternate_value and set VAR= alternate_value

${VAR:=alternate_value}

If VAR defined, use its value else use alternate_value${VAR:-alternate_value}

resultformat

Introduction to Unix 45

More on Variables

Example bash uses of conditional variables:

prompt> echo $VAR $ERROR

prompt> echo ${ERROR:?”An error was found”}bash: ERROR: An error was foundprompt> ERROR=TRUE

prompt> echo ${ERROR:+”An error was found”}An error was found

prompt> echo ${ERROR:-”An error was found”}TRUE

prompt> echo ${ERROR:=”An error was found”}TRUE

Introduction to Unix 46

Compressing and Archiving

There are several ways you can compress files to reduce disk usage or transfer time.

The “Windows” way is using the zip and unzip commands:prompt> ls -alh testfile1 testfile2-rw-r--r-- root root 1.0M testfile1-rw-r--r-- root root 1.0M testfile2

prompt> zip testfile.zip testfile1 testfile2

prompt> ls -alh testfile.zip-rw-r--r-- root root 2.4K testfile.zip

Introduction to Unix 47

Compressing and Archiving

Traditional UNIX archiving tools, tar and gzip.

tar takes a number of files/directories and combines them into a single file

gzip takes combined archive and compresses it:

tar -c file1 file2 ... | gzip -9 > archive.tgz

Or simply:tar -zc archive.tgz file1 file2 …

To extract files from a tar archive:tar xfzp archive.tgz

Introduction to Unix 48

Getting Help from ACCRE

ACCRE website FAQ and Getting Started pages:

www.accre.vanderbilt.edu/support

ACCRE helpdesk:

www.accre.vanderbilt.edu/support/contact/submit_RT.php

accre-forum mailing list

Office hours at ACCRE M-F 4-5 PM

Introduction to Unix 49

Give UNIX a try!

There are several free flavors of UNIX available:•Fedora: http://www.fedora.redhat.com•Debian: http://www.debian.org•FreeBSD: http://www.freebsd.org•Ubuntu: http://www.ubuntu.com

O’Reilly has excellent desktop reference materials (I. e., books):

http://www.oreilly.com/

(E. g., “Linux in a Nutshell”, “Class Shell Scripting”, …)

O’Reilly pocket guides also very useful quick references (Linux OS, shells, editors, …)

\

Introduction to Unix 50

Screen

Another useful unix tool with remote session Tutorial :

… Log onto cluster noting the gateway (i. e., vmpsxx) …prompt> screenprompt> nano junk.txt… While in editor, close window …

… Must log back onto node you were on, i. e., vmpsxx.accre.vanderbilt.edu …

… Find SCREEN process ID (PID) …prompt> ps augx | grep <your user name>prompt> screen -R <PID>… Your process is Restored where you left off …