Lecture 1: Introduction, Basic UNIX

Advanced Programming Techniques

Unix Programming Environment1

Objective: To introduce students to the basic features of Unix and the Unix Philosophy (collection of combinable tools and environment that supports their use) Basic commandsFile systemShellFilters (wc, grep, sort, awk)File redirection, Pipes

Operating Systems

An Operating System controls (manages) hardware and software.provides support for peripherals such

as keyboard, mouse, screen, disk drives, …

software applications use the OS to communicate with peripherals.

The OS typically manages (starts, stops, pauses, etc) applications.

Kernel (OS) Interacts directly with the hardware through device drivers Provides sets of services to programs, insulating these programs from the underlying hardware Manages memory, controls access, maintains file system, handles interrupts, allocates resources of the computer Programs interact with the kernel through system calls

Structure of the UNIX system

Applications

Shell

\Kernel (OS)

Hardware

There are many standard applications:

• file system commands

• text editors

• compilers

• text processing

Unix and Users

Most flavors of Unix (there are many) provide the same set of applications to support humans (commands and shells).Although these user interface programs are not part of the OS directly, they are standardized enough that learning your way around one flavor of Unix is enough.

Flavors of Unix

There are many versions of Unix that are used by lots of people:SysV (from AT&T)BSD (from Berkeley)Solaris (Sun)IRIX (SGI)AIX (IBM)LINUX (free software)

Logging In

To log in to a Unix machine you can either:sit at the console (the computer itself)access remotely, via SSH, e.g.

The system prompts you for your username and password.Usernames and passwords are case sensitive!

CS Dept. Accounts

See http://www.cs.drexel.edu/~kschmidt/Ref/csLogin.htmlAll CS machines (that you have access to) running Linuxtux machines – the farm you may

connect to from anywheretux.cs.drexel.edu

lab machines – any of the desktops you may sit at in the lab, classrooms

Not administered by Drexel IRT

Username

(typically) a sequence of alphanumeric characters of length no more than 8.the primary identifying attribute of your account.(usually) used as an email addressthe name of your home directory is usually related to your username.

Passworda password is a secret string that only the user knows (not even the system knows!)When you enter your password the system encrypts it and compares to a stored string.passwords should have at least 6 charactersIt's a good idea to mix case, include numbers and/or special characters (don't use anything that appears in a dictionary!)

User's Home Directory

The user’s personal directory. E.g.,All home (users') directories (on tux)

are in /home/home/kschmidt/home/vzaychik

Where all your files go (hopefully organised into subdirectories)Mounted from a file server – available (seemlessly) on *any* department machine you log into

Home Directory

Your current directory when you log incd (by itself) takes you homeLocation of many startup and customization files. E.g.:.vimrc .bashrc .bash_profile .forward .pl

an .mozilla/ .elm/ .logout

Files and File Names

A file is a basic unit of storage (usually storage on a disk).Every file has a name.Filenames are case-sensitive!Unix file names can contain any characters (although some make it difficult to access the file) except the null character and the slash (/).Unix file names can be long!how long depends on your specific flavor of Unix

Directories

A directory is a special kind of file - Unix uses a directory to hold information about other files.We often think of a directory as a container that holds other files (or directories).A directory is the same idea as a folder on Windows.

More about File Names

Every file has a name (at least one)Each file in the same directory must have a unique name.Files that are in different directories can have the same name.Files that start with a '.' are, by default, hidden by many utilities

The Filesystem (eg)

/

bin etc home/ tmp usr

hollid2 scully bin etc

netprog unix X ls who

Unix Filesystem

The filesystem is a hierarchical system of organizing files and directories.

The top level in the hierarchy is called the "root" and holds all files and directories in the filesystem.

The name of the root directory is /

Pathnames

The pathname of a file includes the file name and the name of the directory that holds the file, and the name of the directory that holds the directory that holds the file, and the name of the … up to the rootThe pathname of every file in a given filesystem is unique.

Pathnames (cont.)

To create a pathname you start at the root (so you start with "/"), then follow the path down the hierarchy (including each directory name) and you end with the filename.In between every directory name you put a "/".

Pathname Examples

/

bin/ etc/ home/ tmp/ usr/

Hollid2/ scully/ bin/ local/

netprog unix/ X ls who

/usr/bin/lsSyllabus

/home/hollid2/unix/Syllabus

Absolute Pathnames

The pathnames described in the previous slides start at the root.These pathnames are called "absolute pathnames".Special absolute:~kschmidt/ – /home/kschmidt (for

users’ home directories only)~/ – Your home directory (so, relative

to login, $USER)

Relative PathnamesPrefixed w/the current directory, $PWDSo, relative to the current working directory$ cd /home/hollid2$ pwd/home/hollid2$ ls unix/Syllabusunix/Syllabus$ ls Xls: X: No such file or directory$ ls /home/scully/X/home/scully/X

Special Relative paths…

. – The current directory .. – The parent directory$ pwd

/home/holid2

$ ls ./netprog

./netprog

$ ls ../scully

X

Disk vs. FilesystemThe entire hierarchy can actually include many disk drives.some directories can be on other computers

/

bin etc users tmp usr

hollid2 scully

Commands – Basic Syntax

Shell expects the first token (for now) to be a command. Subsequent tokens are argumentsArguments that start with a – are called options (generally, Posixly)

ls -o /home/kschmidt /home/kschmidt/Public

– -o provides a long listing– The final 2 tokens are directories

to be listed

Commands for Traversing Filesystem

ls – lists contents of a directory-a – all files-l – long listing

pwd – print working (current) directorycd – change directoryw/out argument, takes you home

man Pages

To get information about anything that's been properly installed, use man:man lsman catman man

– You can do a keyword search:man -k keyword

Linux boxes also have info pages

The ls command

The ls command displays the names of the named filesGive it the name of a directory as a command line argument to list all the (unhidden) files in the directory.By itself, it lists the current working directory

Command Line Options

We can modify the output format of the ls program with a command line option.The ls command support a bunch of options: -l long format (include file times, owner and

permissions). Compare to -o -a all (shows hidden* files as well as regular

files) -F include special char to indicate file types. -C place into columns

*hidden files have names that start with "."

cd – change directory

The cd command can change the current working directory:cd change directory

The general form is:cd [directoryname]

By itself, returns you to your home directory

Viewing files

cat – concatenate, send to stdout. View contents of text filesless, more – paging utilities (hit ‘h’ for help, 'q' to quit)od – octal dump. For viewing raw data in octal, hex, control chars, etc.

Copying, removing, linking

rm – remove filerm ~/tmp/download

mv – move (rename) filemv old.file ../otherDir/new.name

cp – copy filecp someDir/file someDir/file.copy

ln – create hard (inode) or soft (symbolic) links to a file

Commands for directories

mkdir make directoryrmdir remove directoryDirectories can also be moved or renamed (mv), and copied (cp –r)

Commands for Archiving

tar – Tape Archivemakes a large file from many files

gzip, gunzipcompression utility

tar on Linux does gzip compression with the z option:$ tar czf 571back.tgz CS571

$ tar xzf assn1.tgz

File attributes

Every file has some attributes:Access Times:

when the file was createdwhen the file was last changedwhen the file was last read

SizeOwners (user and group)PermissionsType – directory, link, regular file, etc.ACLs – access control lists (not today)

File Time Attributes

Time Attributes:when the file was last changed ls -lsort by modification time ls -lt

File Owners

Each file is owned by a user.You can find out the username of the file's owner with the -l or -o option to ls:

[jjohnson@ws44 winter]$ ls -l

total 24

drwxr-xr-x 7 jjohnson users 80 Jan 3 2005 cs265/

-rw------- 1 jjohnson users 8258 Jan 3 2005 cs265.html

-rw-r--r-- 1 jjohnson users 8261 Jan 3 2005 cs265.html~

ls -l

$ ls -l foo

-rw-rw---- 1 hollingd grads 13 Jan 10 23:05 foo

permissionsowner group

size

time

name

File Permissions

Each file has a set of permissions that control who can mess with the file.There are three types of permissions:

read abbreviated r write abbreviated w execute abbreviated xThere are 3 sets of permissions:

1. user2. group3. other (the world, everybody else)

ls -l and permissions

-rwxrwxrwx User Group Others

Type of file:- – plain filed – directorys – symbolic link(others)

rwx

Files:r - allowed to read.w - allowed to writex - allowed to execute

Directories:r - allowed to see the names of the

contentsw - allowed to add and remove files.x - allowed to “enter” the directory

Changing Permissions

The chmod command changes the permissions associated with a file or directory.There are a number of forms of chmod, this is the simplest:

chmod mode file

chmod – numeric modesConsider permission for each set of users (user, group, other) as a 3-bit # r – 4w – 2x – 1

A permission (mode) for all 3 classes is a 3-digit octal #755 – rwxr-xr-x644 – rw-r—r--700 – rwx------

chmod - examples$ chmod 700 CS571$ ls –o Personaldrwx------ 10 kschmidt 4096 Dec 19 2004 CS571/

$ chmod 755 public_html$ chmod 644 public_html/index.html$ ls –ao public_htmldrwxr-xr-x 16 kschmidt 4096 Jan 8 10:15 .drwx--x--x 92 kschmidt 8192 Jan 8 13:36 ..-rw-r--r-- 5 kschmidt 151 Nov 16 19:18 index.html

$ chmod 644 .plan$ ls –o .plan-rw-r--r-- 5 kschmidt 151 Nov 16 19:18 .plan

chmod – symbolic modes

Can be used to set, add, or remove permissionsMode has the following form:

[ugoa][+-=][rwx]u – user g – group o – other a – all+ add permission - remove permission

= set permission

chmod examples

$ ls -al foo

-rwxrwx--x 1 hollingd grads foo

$ chmod g-wx foo

$ ls -al foo

-rwxr----x 1 hollingd grads foo

$ chmod u-r .

$ ls

ls: .: Permission denied

Shell as a user interface

A shell is a command interpreter, an interface between a human (or another program) and the OSruns a program, perhaps the ls

program.allows you to edit a command line.can establish alternative sources of

input and destinations for output for programs.

Is, itself, just another program

Bourne-again Shell (bash)

We’ll teach bash in this courseExtension of the Bourne Shell (sh)Contains many of the Korn Shell (ksh) extensionsYou may use the shell of your choice (tcsh, zsh, etc.), but that’s on you.

Session StartupOnce you log in, your shell will be started and it will display a prompt.(for our examples, we will use $ as

the prompt. It is not part of the input)

When the shell is started it looks in your home directory for some customization files.You can change the shell prompt,

your PATH, and a bunch of other things by creating customization files.

Customization

Each shell supports some customization.User promptWhere to find mailShortcuts

The customization takes place in startup files – files that are read by the shell when it starts up

Startup filessh,ksh:

/etc/profile (system defaults) ~/.profile

bash:

~/.bash_profile

~/.bashrc

~/.bash_logout

csh:

~/.cshrc

~/.login

~/.logout

Incorrect login

You will receive the “Password:” prompt even if you type an incorrect or nonexistent login name– Can you guess why?Nothing will happen while you type your password. It's fine

Entering Commands

The shell prints a prompt and waits for you to type in a command.The first token on the line is taken to be a command (for now). Come in 2 flavors:shell builtin - commands that the shell

interprets directly.External programs (utilities) –

standalone programs on disk (directories in your $PATH are searched, in order)

Interpreting a Command - type

When a command is seen, the shell:1. Checks for aliases2. Checks for user-defined functions3. Looks for a builtin4. Checks directories in $PATH for a utility

Use Bash’s type builtin to see what the shell is using:kschmidt@ws60 kschmidt> type echoecho is a shell builtinkschmidt@ws60 kschmidt> type chmodchmod is /bin/chmod

Command Options and Arguments

standardized command syntax (applies to most commands):

command option(s) arguments

options modify the way in which a command works, often single letters prefixed with a dash (can be sometimes combined after a single dash

Getting helpmanual – original Unix help (flat, single page)$ man bashinfo – 2-d system, emacs-like navigation$ info bashUse the help builtin for help on Bash builtin commandsThe resource frame on the class pageInternet – google, wikipedia

The linux documentation project (http://www.tldp.org/)Safari online Friends, group-mates, and others

Some simple commandsdate – print current datewho – print who is currently logged infinger user – more information about userls -ao – lists (long) all files in a directorydu -sh – disk usage summary, human readablequota

Logging off

exit builtin (command)Exits the shellIf it is the login (top-level) shell, then

it disconnects you

A shell is just another program that is running. Can recursively invoke shellsPlease don’t just disconnect w/out exiting

Standard I/OWhen you enter a command the shell creates a subshell to run the process or script.The shell establishes 3 I/O channels:Standard Input (0) – keyboardStandard Output (1) – screenStandard Error (2) – screen

These streams my be redirected to/from a file, or even another command

Programs and Standard I/O

ProgramProgramStandard Input

(STDIN)Standard Output

(STDOUT)

Standard Error(STDERR)

Terminating Standard Input

If standard input is your keyboard, you can type stuff in that goes to a program.To signal the end of input press Ctrl-D (^D), the EOF signal, on a line by itself, this closes the input stream.The shell is a program that reads from standard input.What happens when you give the shell ^D? (See the bash set command, ignoreeof)

Shell metacharactersSome characters have special meaning to the shell. These are just a few:I/O redirection< > |

wildcards* ? [ ]

others& ; $ ! \ ( ) space tab newline

These must be escaped or quoted to inhibit special behavior

Wildcards* – matches 0 or more characters? – matches exactly 1 character[<list>] – matches any single character in <list>

E.g.ls *.cc – list all C++ source files in directory

ls a* – list all files that start w/’a’ls a*.jpeg – list all JPEGs that start w/’a’ls * - (make sure you have a subdirectory, and try it)

Wildcards (more examples)

ls file?

- matches file1, file2, but not file nor file22

ls file?.*.DEL

- matches file1.h.DEL, file9.cc.DEL, file3..DEL but not file8.DEL nor file.html.DELThese are not regular expressions!

Wildcards - classes[abc] matches any of the enclosed

charactersls T[eE][sS][tT].doc

[a-z] matches any character in a rangels [a-zA-Z]*

[!abc…] matches any character except those listed.

ls [!0-9]*

Shell Variables

bash uses shell variables to store informationShell variables are used to affect the behavior of the shell, and many other programsWe can access these variables:set new values for some to customize

the shell.find out the value of some to help

accomplish a task.

Setting/Viewing VariablesTo assign (in sh, ksh, bash):VAR=someStringOTHER_VAR=“I have whitespace”Note, no whitespace around the ‘=‘!

To view (dereference) a variable:$ echo $VARsomeString$ echo $OTHER_VARI have whitespace

Shell maintains some variables

Some common ones:PATH – list of directories shell searches

for non-shell commandsPS1 – Primary promptUSER – user's login nameHOME – user’s home directoryPWD – current working directory

Other Useful OnesSHELL – the login shellTERM – the type of terminal

interfaceHISTFILE – where your command

history is savedEDITOR – holds user's preferred

editorHOSTNAME – machine's hostnameSHELLOPTS – status of various shell

options (see Bash's set built-in)

Displaying Shell Variables

Prefix the name of a shell variable with "$".The echo command will do:$ echo $HOME

$ echo $PATH

You can use these variables on any command line:

$ ls -al $HOME

Setting Shell Variables

You can change the value of a shell variable with an assignment command (this is a shell builtin command):

HOME=/etc

PATH=/usr/bin:/usr/etc:/sbin

NEWVAR="blah blah blah"

set command (shell builtin)

The set command with no args prints out a list of all the shell variables.Some bash optionsnoclobber – won't let re-direct

overwrite an existing fileignoreeof – Shell won't exit on ctrl-Dvi – use vi-style interface (by default,

bash uses emacs-style key bindings)-n – dry-run (just parse, but don't

execute). Handy for debugging scripts

Quoting – escape character, \

Use the backslash to inhibit the special meaning of the following character:$ echo $USER

kschmidt

$ echo \$USER

$USER

$ echo a\\b

a\b

Quoting – double quotes

Double quotes inhibit all behavior except variable substitution, command substitution, and the escape, \$ echo “$USER is $USER”

kschmidt is kschmidt

$ echo “\$USER is $USER”

$USER is kschmidt

$ echo “I said, \”Wait a moment\””

I said, “Wait a moment”

Quoting – single quotes

Single quotes inhibit nearly all special behaviorMay not contain a single quote$ echo ‘I said “Wait!”’

I said “Wait!”

$ echo ‘My name is $USER’

My name is $USER

$ mv rambleOnByLedZeppelin ‘ramble on – led zeppelin’

Input Redirection

The shell can read stdin from places other than your keyboard, like, a disk file, or even output from another commandA file (the contents of the file are fed

to a program as if you typed it).A pipe (the output of another program

is fed as input as if you typed it).

Output Redirection

The shell can attach things other than your screen to standard output (or stderr).A file (the output of a program is

stored in file).A pipe (the output of a program is fed

as input to another program).

Redirecting stdout

Use “>” after a command (and its arguments) to send output to a file:ls > lsoutif lsout previously existed it will be

truncated (gone), unless noclobber is set (see bash)

Redirecting stdin

To tell the shell to get standard input from a file, use the “<“ character:

sort < nums– Sort the lines in the file nums ,

sends the result to stdout.

You can do both!

sort < nums > sortednums

tr a-z A-Z < letter > rudeletter

Appending Output

Use >> to append append output to a file:ls /etc >> foo

ls /usr >> foo

Easy way to concatenate files:cat rest_of_file >> my_file

Redirecting stderrstderr is file descriptor 2, so:

gcc buggy.c 2> error.log

grep ‘[Vv]era’ *.html > log 2> errorlog

To send both to the same place (stdout is file descriptor 1):

find . -name 'core*' > core.lis 2>&1

Orfind . -name 'core*' 2> core.lis

Pipes – connecting processes

A pipe is a holder for a stream of data.A pipe can be used to hold the output of one program and feed it to the input of another.

prog1prog1 prog2prog2STDOUT STDIN

Asking for a pipe

Separate 2 commands with the “|” character.The shell does all the work!

ls -1 | sort

ls -1 | sort > sortedlist

ls -1 | sort | head > top.ten

The Unix Philosophy

Stringing small utilities together with pipes and redirection to accomplish non-trivial tasks easilyE.g., find the 3 largest subdirectories:$ du –s * | sort –nr | head -n3120180 Files

22652 Zaychik

9472 tweedledee.tgz

filtersPrograms that read some input (but don’t change it), perform a simple transformation on it, and write some output (to stdout)Some common filters…wc – word count (line count, character

count)tr – translategrep, egrep – search files using regular

expressionssort – sorts files by line (lexically or

numerically)cut – select portions of a lineuniq – Removes identical adjacent lineshead, tail – displays first (last) n lines of a

file

pipes and combining filters

Connect the output of one command to the input of another command to obtain a composition of filters

who | wc -lls | sort -fls -s | sort -nls -l | sort -nr -k4ls -l | grep ‘^d’

Process ControlProcesses are run in a subshell (by default)Subshells inherit exported variablesEach process is has an ID (pid) and a parent (ppid)Use the ps utility to look at some processes:$ ps

PID TTY TIME CMD 350 pts/4 00:00:00 bash22251 pts/4 00:00:00 vim22300 pts/4 00:00:00 ps

Process Control (cont.)

Use the –f option for a long listing:$ ps –f

UID PID PPID C STIME TTY TIME CMD

kschmidt 350 349 0 10:06 pts/4 00:00:00 -bash

kschmidt 22251 350 0 17:32 pts/4 00:00:00 vim myHomework

kschmidt 22437 350 0 17:36 pts/4 00:00:00 ps -f

Use the –e option to see more processes (all of them).$ ps –e | grep xmms

29940 pts/0 00:33:47 xmms

Killing a process (not usually nice)

The kill command sends a signal to a process (the given pid)By default, sends TERM (terminate), which asks the process to finish, so that it may do clean-upuse -9 to send a KILL (won’t be ignored), but no cleanupMy mp3 player hangs once in while:$ kill -9 29940

Job Control

The shell allows you to manage jobsplace jobs in the backgroundmove a job to the foregroundsuspend a jobkill a job

Background jobsIf you follow a command line with "&", the shell will run the job in the background.you don't need to wait for the job to

complete, you can type in a new command right away.

you can have a bunch of jobs running at once.

you can do all this within a single terminal (window).

ls -lR > saved_ls &

Listing jobs

The command jobs will list all background jobs:

> jobs

[1] Running ls -lR > saved_ls &

>

The shell assigns a number to each job (this one is job number 1).

Suspending and Resuming the Foreground Job

You can suspend the foreground job by pressing ^Z (Ctrl-Z). Suspend means the job is stopped, but not dead. The job will show up in the jobs output.

You give fg a job number (as reported by the jobs command) preceeded by a %. Without an argument, fg brings the last job forward

$ jobs[1] Stopped ls -lR > saved_ls &$ fg %1ls -lR > saved_ls

Placing a suspended job in the background

If it’s in the foreground, suspend itUse bg, just as you did fg, to let a suspended job continue in the background:$ bg %3

Killing a job

Kill may also take a job number or even a job name, introduced by %:$ find . –name core\* -print > corefiles &

$ firefox&

$ jobs

[1]+ Running find . –name …

[2]+ Running firefox

$ kill %2

EditorsA text editor is used to create and modify text files.The most commonly used editors in the Unix community:vi (vim on Linux)

$ vimtutor

emac $ emacs Then, hit ctrl-h t (that’s control-h, followed by ‘t’)

You must learn at least one of these editors