Intermediate UnixIntermediate Unix

Presented July 29th, 2001 by:

“Robin” R. Battey (zanfur@cs.washington.edu)

Evgeny Roubinchtein (evgenyr@cs.washington.edu)

with tips, suggestions, and corrections by:

Hannah Tang (hctang@cs.washington.edu)

http://www.cs.washington.edu/people/acm/tutorials/

The “File System”The “File System”

• Under UNIX, (almost) everything is a “file”:– Normal files– Directories– Hardware– Sockets– Pipes

• Things that are not files:– Users– Groups– Processes

OwnershipOwnership• Files have two owners

– Every file has exactly one user owner– Every file has exactly one group owner

• Everyone is a user– Users are in at least one group

• Processes have owners, too (known as an “id”)– Every process has exactly one user id– Every process has at least on group id

• Users and groups are really just numbers with names– Every username is mapped to a single numeric “uid”– Every groupname is mapped to a single numeric “gid”

Who am I?Who am I?

• Commands that tell you who you are:– whoami displays your username– id displays your username and groups

• Commands that tell you who others are:– finger [<name>] displays info for <name>– id [<username>] displays info for <username>

• Commands that change who you are:– su <username> “switch user” to <username> – login login as a different user

File PermissionsFile Permissions

• Every file has three access levels:– user (the user owner of the file)

– group (the group owner of the file)

– other (everyone else)

• At each level, there are three access types:– read (looking at the contents)

– write (altering the contents)

– execute (executing the contents)

Strange ThingsStrange Things

• There are three “strange” permissions:– setuid (run program as user owner)– setgid (run program as group owner)– text (stay in swap after executing)

• Directories act differently– “write” (creating/deleting files)– “execute” (cd-ing to that directory)– “setuid” (ignored)– “setgid” (created files have same group owner)– “text” (deletion restricted to user owned files)

Examining PermissionsExamining Permissions

• A “long” ls listing shows file permissions:

[zanfur@odin zanfur]$ iduid=8774(zanfur) gid=100(users) groups=100(users),101(mp3)[zanfur@odin zanfur]$ ls -ltotal 524-rwxr-xr-x 1 zanfur users 512668 Jul 31 00:18 bashprw-r--r-- 1 zanfur users 0 Jul 31 00:24 fifo-rw-r--r-- 1 zanfur users 0 Jul 31 00:18 filedrwxr-xr-x 2 zanfur users 4096 Jul 31 00:16 normaldrwxrws--T 2 zanfur mp3 4096 Jul 31 00:14 shareddrwxrwxrwt 2 zanfur users 4096 Jul 31 00:14 tmpdrwxrwxr-x 2 zanfur www 4096 Jul 31 00:15 www[zanfur@odin zanfur]$ _

Permissions Listing BreakdownPermissions Listing Breakdown

Field Description Valid Values

?--------- file type - (normal file), d (directory), p (fifo), b, c, s

-?-------- user read - (no read permissions), r (read permissions)

--?------- user write - (no write permissions), w (write permissions)

---?------ user execute - (no exec), x (exec), S (setuid), s (both)

----?----- group read - (no read permissions), r (read permissions)

-----?---- group write - (no write permissions), w (write permissions)

------?--- group execute - (no exec), x (exec), S (setgid), s (both)

-------?-- other read - (no read permissions), r (read permissions)

--------?- other write - (no write permissions), w (write permissions)

---------? other execute - (no exec), x (exec), T (text), t (both)

What You Can Do With PermissionsWhat You Can Do With Permissions

Permission File Directory

r (read) Read a file List files in …

w (write) Write a file Create a file in …

Rename a file in …

Delete a file in …

x (execute) Execute a file (eg shell script)

Read a file in …

Write to a file in …

Execute a file/shell script in …

Changing OwnershipChanging Ownership

• Changing user ownership:– The command is “change owner”:

chown <username> <filename>– but, you can only do it if you are root– so just copy it instead

• Changing group ownership:– The command is “change group”:

chgrp <groupname> <filename>– but, you can only do it if you are in group <groupname>

– and you must be the user owner of the file

Changing PermissionsChanging Permissions• The “change mode” command:

chmod <level><op><permissions>[,…] <filename>

<level> string of: u, g, o, a (user, group, other, all)<op> one of +, -, = (gets, loses, equals)<permissions> string of: r, w, x, s, t, u, g, o

(read, write, execute, set-id, text,same as user, same as group, same as

other),

• Examples:chmod u+rwx,go-w foobarchmod g=u,+t temp/chmod u=rwx,g=rwxs,o= shared/

Process ManagementProcess Management

• What can you do with it?– Start programs in the background– Run more than one program per terminal– Kill bad and/or crashing programs– Suspend programs mid-execution– List all jobs running in a shell– Move foreground jobs to the background– More …

Three States of a ProcessThree States of a Process• Foreground

– Attached to keyboard– Outputs to the screen– Shell waits until the process ends

• Background, running– Not attached to keyboard– Might output to the screen– Shell immediately gives you another prompt

• Background, suspended– Paused mid-execution– Can be resumed in background or foreground

Background ProcessesBackground Processes• Listing jobs:

– jobs lists background “jobs” and job #’s– ps lists processes and their process id (“pid”)– %<job#> expands to the process id of the job

• Stopping foreground jobs– Press ^Z (Ctrl-Z) in the terminal window

• Starting a process in the background– Append a & character to the command line– Examples: ls –lR > ls-lR.out &

xemacs my_program.cc &

• Resuming a stopped job– In the foreground: fg [<pid>]– In the background: bg [<pid>]

Killing ProcessesKilling Processes• The “kill” command:

kill [-<signal>] <pid>Send <signal> to process <pid>

• The “killall” command:killall [-<signal>] <command>Send <signal> to all processes that start with <command>

• Useful signals (kill –l for the complete list):TERM the default, “terminate”, kills things nicelyKILL will kill anything, but not nicelyHUP “hangup”, used to reload configurationsSTOP stops (suspends) a running process

What are environment variables?What are environment variables?

• The environment is an array of strings of the formNAME=VALUE

• Each process (program) has its own copy of the environment

• Processes started by the shell get a (“deep”) copy of the shell’s current environment

• Each process can examine and modify its own (and only its own) environment

• The “meaning” of environment variables is merely a matter of established conventions

Viewing your shell’s environmentViewing your shell’s environment

• To view a single environment variable’s value:echo $<name>

• For example:echo $HOME

will display/homes/iws/evgenyr

• To view all environment variables at once:– tcsh/csh/bash: printenv– sh/ksh: set

Setting environment variablesSetting environment variables

• tcsh/csh:– setenv <name> <value>– Example: setenv PRINTER ps329

• bash/ksh:– export <name>=<value>– There is no space on either side of ‘=’!

– Example: export PRINTER=ps329

• sh:– <name>=<value>; export <name>

!

Appending to environment variablesAppending to environment variables

• Appending /uns/bin/ to your PATH– bash/ksh: export PATH=$PATH:/uns/bin

– tcsh/csh: setenv PATH=$PATH:/uns/bin

• Prepending is similar:– bash/ksh: export INFOPATH=/uns/info:$INFOPATH

– tcsh/csh: setenv INFOPATH /uns/info:$INFOPATH

Common environment variablesCommon environment variables

• The “meaning” of environment variables is purely a matter of convention.

• However, these environment variables are quite common:– PATH, INFOPATH, MANPATH, EDITOR, VISUAL, PAGER, HOME, MAIL, USER

• The man page for a program will usually list the environment variables the program pays attention to.

Useful shell features Useful shell features (a preview of upcoming attractions)(a preview of upcoming attractions)

• Aliases

• Redirecting input and output

• Command substitution

• Scripts

Shell as a work-saver: aliasesShell as a work-saver: aliases• Aliases: textual substitution, similar to

C-preprocessor macros.• Syntax:

– bash/ksh: alias <text>=’<replacement>’

– tcsh/cshalias <text> ’<replacement>’

• When you type <text>, the shell “substitutes” <replacement>

• Typing alias by itself lists all your current aliases• unalias <text> removes the alias• Check the csh/tcsh man page for extra features

AliasAlias examples examples

• Always print postscript files double-sided– tcsh/csh:alias lpr ’lpr –Zduplex’

– bash/ksh: alias lpr=’lpr –Zduplex’

• Collect a few tree-friendly options to enscript:– tcsh/csh:alias print \ ’enscript –2rhB –SDuplex=DuplexNoTumble’

Line continuation character, just like in C/C++!

What is input/output redirection?What is input/output redirection?

• Normally, a program’s standard output is displayed on user’s terminal, and its standard input comes from the keyboard.

• Redirecting the output of a program means asking the shell to put the program’s output (stdout [C++’s cout]) into a file.

• Redirecting the input of a program means asking the shell to feed a file as the program’s standard input (stdin [C++’s cin]).

• Note: redirection works with files.

Why redirect program’s output?Why redirect program’s output?

• You may want to save output and examine it at your leisure:– if the program generates a lot of output

– if the program takes a long time to run

• You may want to present the output to another person

• Having the program write to standard output may make the program simpler to write

Standard output vs Standard errorStandard output vs Standard error

• By convention, “normal” output of a program is sent to standard output (stdout [C++’s cout]), while debugging or error output is sent to standard error (stderr [C++’s cerr]).

How to redirect program’s output?How to redirect program’s output?• To redirect just the standard output:<program> > <FILE>

• To redirect just the standard error:sh/ksh/bash: <program> 2> <FILE>csh/tcsh: ( <program> > STDOUT ) >& STDERR

• To redirect both standard output and standard error:

csh/tcsh/bash: <program> >& <FILE>sh/ksh/bash: <program> > <FILE> 2>&1

> vs. >>> vs. >>

• Both > and >> will create the output file, if it doesn’t already exist

• If the file does exist, then:– Using > to redirect output will overwrite the output

file:•ls > newlisting•printenv > my_environment

– Using >> to redirect output will append to the output file•cat ch1 ch2 ch3 > book•cat ch4 ch5 ch6 >> book

Why redirect program’s input?Why redirect program’s input?

• To run the program repeatedly with the same (or similar input)

• Having the program read from standard input may make the program simpler to write.

How to redirect program’s input?How to redirect program’s input?

• Simple!<program> < <FILE>

• Examplesort < my_grades.txthead < really_long_book.txt

PipingPiping

• Piping is connecting programs together by using the output of one program as the input to the next.

• Syntax:<program1> | <program2> | … | <programN>

• A simple example (view a sorted file-listing a page at a time):ls | sort | less

• Note: piping deals with the input/output of programs (that is, stdin, stdout, stderr)

Why piping?Why piping?

• Because “the whole is bigger than the sum of its parts.

• By combining Unix utilities in a pipeline, you can build tools “on-the-fly” as you need them.

Piping examplesPiping examples

• How many .c files are in this directory?ls *.c | wc –l

• What files were modified most recently?ls –t | head

• What processes am I running?ps auxw | grep <mylogin>

• Make an alias for the above, for other Linux boxen too:alias myps=’ ps auxw | grep `id –un`’

TheThe catcat utilityutility

• Especially useful: cat– When used with pipes, “copies” stdin to stdout

•cat can be used to redirect out of a pipe!

• Example: make a file containing currently running processes ps aux | grep evgenyr | cat > my_processes

– When used with a file, “copies” the file into stdout•cat can be used to place a file’s contents into a pipe!

• Example: list all the items in a list in alphabetical ordercat my_grocery_list.txt | sort | uniq

Command substitutionCommand substitution(aka “what’s up with the backquotes?”)(aka “what’s up with the backquotes?”)

• Command substitution means that the shell substitutes a command’s output for the command itself.

• To get the shell to perform command substitution, enclose the command in backquotes (`)

Shell as a work-saver: scriptsShell as a work-saver: scripts• Instead of typing the same series of commands over and

over again, put them in a file and have the shell execute the (commands in the) file!

• The file must have execute permission• The first line of the file should be:#! <YOURSHELL> (e.g., /bin/bash)

• There must be no space (or any other character) between ‘#’ and ‘!’.

• Whether to put space after ‘!’ is a matter of style• Shell also has control flow, tests, etc. The shell tutorial

on the ACM web page goes into much more detail.

Intermezzo: quotingIntermezzo: quoting

• Problem: some characters are special to the shell (*, ?), but you would like to pass those characters to the programs the shell runs as-is.

• So you quote the character(s):– A single character: by prefixing it with a \ (backslash)

– A string of characters: by enclosing them in• Single quotes: no characters are special. None

• Double quotes: some characters (notably $ and `) are still special. You’ll need to prefix those with a backslash to pass to the program as-is.

Quoting examplesQuoting examples• List all the files in the current directory:ls *

• List the file or directory called ‘*’ (or complain if it’s not there):ls \*

• Print $HOME:echo ’$HOME’

• Print the path to your home directory:echo ”$HOME”

• Print `id –un`: echo ’`id –un`’

• Print your login:echo ”`id –un`”

Next stop: utilitiesNext stop: utilities

• (No, not electricity, water, and sewer)• diff and patch• grep• find and xargs

diffdiff and and patchpatch

• You have two versions of a file; how do you see what’s changed between the two versions?– diff shows the differences between two files; you’ll

want to use the –u or –c option to diff to produce output in human-friendly format:diff –u my_file my_file.orig | less

– patch applies differences to a filediff –u my_file my_file.orig > my_patchpatch my_file < mypatch

my_file is now the same as my_file.orig

grep grep – search for patterns– search for patterns

• grep searches for patterns in texts:grep <string> <FILE>

• grep uses regular expressions to describe the string(s) to search for

• In a regular expression, most characters are treated as-is, but a few are magic– Ordinary characters just represent themselves– Magic characters do special things

grepgrep’s magic characters’s magic characters

• A few different kinds of magic characters:– Some characters “anchor” (parts of) a regular

expressions to specific places in the string:^ - The beginning of a line$ - The end of a line

– A dot (.) matches “any one character whasoever” (except for newline)

– [ ] form a character class:•[aeiou] – any single vowel•[a-zA-z0-9] – Any single letter or digit•[^0-9] – Any single character that isn’t a digit

Here, ‘^’means “not”

Even more of Even more of grepgrep’s magic characters’s magic characters

• Quantifiers say how many times the preceeding “thing” should be repeated:– * means “zero or more times”– ? Means “zero or one time”

Frequently used Frequently used grepgrep options options

-i : do case-insensitive search

-n : print line numbers

-v : print lines that do not match

-l : only list the files from which output would have been printed

grep grep examplesexamples

• Print all non-blank lines:grep –v ’^$’ my_file.txt

• Print all the lines on which my_function is called (or declared):grep –n ’my_function *(’ my_code.c

• Show all the xterms I am running:ps auxw | grep ”`id –un` .*xterm”

findfind

• Traverse the tree(s) rooted at <PATHs> , and for each “thing” found, evaluate the <EXPRESSION> until the result of <EXPRESSION> is known.

• The evaluation of <EXPRESSION> “short-circuits”, just like expressions in C/C++.– false && some_expression can never be true

• Options apply to the entire find command, not the individual expression

find <PATHS> <OPTIONS> <EXPRESSION>

Components of a Components of a findfind expression expression

• An expression is zero or more primaries connected by operators

• Two kinds of primaries:– Tests: just return true or false– Actions: do something, in addition to returning true or false– Operators work just as they do in C/C++.

•! / -not•-a / -and•-o / -or•, (comma)

– Parentheses are used for grouping, just as in C/C++.

findfind examples examples• Print all the *.c* and *.h* files in and below the current directory

find . –name ’*.[ch]*’ –a –print

• Show line numbers myfunction calls in the above *.c* filesfind . –name ’*.c*’ | xargs grep –n ’myfunction.*(’

• Change permissions on files under your home directory so they’re inaccessible to everyone but you (except for files in the www directory)cd; find . –path ”./www*” –prune –o –exec chmod go-rwx {} \;

• How big are my *.c* files?expr `find –name ’*.c*’ –printf ”%k + ”` 0

• Read the find manual (info find) for more examples

xargsxargs: combine arguments: combine arguments

• Feeds standard input as arguments to <COMMAND>• Often used with find

(find … | xargs grep)• But it doesn’t have to be used with find:cat filelist | xargs cat {} > concatenated

xargs <OPTION> <COMMAND> <INITIAL_ARGS>

Finding more information – at CSEFinding more information – at CSE

• Use info and man• Peruse the “see also” section of the man pages

• Peruse info’s i (search index) command

• The uw-cs.lab-help group

• Look at your .login, .cshrc, etc. to see how support has set things up

Finding more info – on the webFinding more info – on the web• http://www.faqs.org (comp.unix questions FAQ)• http://www.google.com• http://www.deja.com• The support web page:

http://www.cs.washington.edu/lab/ about/ugradcomputing.html

• (Shameless plug) the ACM tutorials page:http://www.cs.washington.edu/people/acm/tutorials/