Formation sh

SyntaxVariables

Built-in CommandsAwk, Sed et Co.

Scripting with Sh

Nicolas Ledez

15 septembre 2008

Nicolas Ledez Scripting with Sh

SyntaxVariables


Outline

1 Syntax

2 Variables

3 Built-in Commands

4 Awk, Sed et Co.


SyntaxVariables


Filename MetacharactersQuotingCommand FormsRedirection Forms

Filename Metacharacters

String Meaning* Match any string of zero or more characters? Match any single character[abc...] Match any one of the enclosed characters ; a

hyphen can specify a range (e.g., a–z, A–Z,0–9)

[!abc...] Match any character not enclosed as above


SyntaxVariables



Filename Metacharacters - examples

String Meaningnew* Match new new.1ch? Match ch9 but not ch10[D-R]* Match files that begin with uppercase D through R


SyntaxVariables



Quoting

Character Meaning; Command separator& Background execution( ) Command grouping| Pipe< > & Redirection symbols* ? [ ] ~ + - @ ! Filename metacharacters" ’ \ Used in quoting other characters‘ Command substitution$ Variable substitution

space tab newline Word separators


SyntaxVariables



Quoting - examples

$ echo ’Single quotes "protect" double quotes’Single quotes "protect" double quotes$ echo "Well, isn’t that \"special\" ?"Well, isn’t that "special" ?$ echo "You have ‘ls | wc -l‘ files in ‘pwd‘"You have 43 files in /home/bob


SyntaxVariables



Command Forms 1/2

cmd & Execute cmd in backgroundcmd1 ; cmd2 Command sequence ; execute

multiple cmds on the same line{ cmd1 ; cmd2 ; } Execute commands as a group

in the current shell(cmd1 ; cmd2) Execute commands as a group

in a subshellcmd1 | cmd2 Pipe ; use output from cmd1 as

input to cmd2cmd1 ‘cmd2‘ Command substitution ; use

cmd2 output as arguments tocmd1


SyntaxVariables



Command Forms 2/2

cmd1 && cmd2 AND ; execute cmd1 and then (ifcmd1 succeeds) cmd2. This is a“short-circuit” operation ; cmd2is never executed if cmd1 fails

cmd1 || cmd2 OR ; execute either cmd1 or(if cmd1 fails) cmd2. This is a“short-circuit” operation ; cmd2is never executed if cmd1 suc-ceeds


Command Forms - examples

$ nroff file > file.txt &Format in the background$ cd ; lsExecute sequentially$ (date ; who ; pwd) > logfileAll output is redirected$ sort file | pr -3 | lpSort file, page output, then print$ vi ‘grep -l ifdef *.c‘Edit files found by grep$ egrep ’(yes|no)’ ‘cat list‘Specify a list of files to search$ grep XX file && lp filePrint file if it contains the pattern$ grep XX file || echo "XX not found"otherwise, echo an error message

SyntaxVariables



Redirection Forms

File De-scriptor

Name CommonAbbreviation

Typical Default

0 Standard input stdin Keyboard1 Standard output stdout Terminal2 Standard error stderr Terminal


SyntaxVariables



Redirection Forms - Simple redirection

$ cmd > fileSend output of cmd to file (overwrite)$ cmd » fileSend output of cmd to file (append)$ cmd < fileTake input for cmd from file$ cmd « textThe contents of the shell script up to a line identical to textbecome the standard input for cmd (text can be stored ina shell variable)


SyntaxVariables



Redirection Forms - Redirection using file descriptors

cmd >&n Send cmd output to file descriptor ncmd m>&n Same, except that output that would normally

go to file descriptor m is sent to file descriptorn instead

cmd >&- Close standard outputcmd <&n Take input for cmd from file descriptor ncmd m<&n Same, except that input that would normally

come from file descriptor m comes from filedescriptor n instead

cmd <&- Close standard input


SyntaxVariables



Redirection Forms - examples 1/2

$ cat part1 > book

$ cat part2 part3 >> book

$ mail tim < report

$ sed ’s/^/XX /g’ << "END_ARCHIVE"> echo toto $titi> END_ARCHIVEXX echo toto $titi

$ export titi=qsdf$ sed ’s/^/XX /g’ << END_ARCHIVE> echo toto $titi> END_ARCHIVEXX echo toto qsdf


SyntaxVariables



Redirection Forms - examples 2/2

$ echo "Usage error: see administrator" 1>&2To redirect standard output to standard error$ find / -print > filelist 2>no_access

The following command sends output (files found) to filelist anderror messages (inaccessible files) to file no_access


SyntaxVariables


Variable SubstitutionBuilt-in Shell VariablesOther Shell Variables

Variable Substitution

String Meaningvar=value Set each variable var to a value$var ${var} Use value of var${var:-value} Use var if set ; otherwise, use value${var:=value} Use var if set ; otherwise, use value

and assign value to var${var:?value} se var if set ; otherwise, print value and

exit (if not interactive). If value isn’tsupplied, print the phrase "parameternull or not set."

${var:+value} Use value if var is set ; otherwise, usenothing


SyntaxVariables



Variable Substitution 1/2

$ u=up d=down blank=Assign values to three variables (last is null)$ echo ${u}rootuproot

Braces are needed here$ echo ${u-$d}up

Display value of u or d ; since u is set, it’s printed$ echo ${tmp-‘date‘}Thu Feb 4 15:03:46 EST 1993

If tmp is not set, the date command is executed


SyntaxVariables



Variable Substitution 1/2

$ u=up d=down blank=

$ echo ${blank="no data"}blank is set, so it is printed (a blank line)$ echo ${blank:="no data"}no data

blank is set but null, so the string is printed$ echo $blankno data

blank now has a new value


SyntaxVariables



Built-in Shell Variables 1/2

String Meaning$# Number of command-line arguments$- Options currently in effect (arguments

supplied to sh or to set)$? Exit value of last executed command$$ Process number of current process$! Process number of last background

command


SyntaxVariables



Built-in Shell Variables 2/2

String Meaning$0 First word ; that is, command name.

This will have the full path name if itwas found via a PATH search

$n Individual arguments on commandline (positional parameters, n = 1–9).

$*, $@ All arguments on command line($1 $2 ...)

"$*" All arguments on command line asone string ("$1 $2...")

"$@" All arguments on command line, indi-vidually quoted ("$1" "$2" ...)


SyntaxVariables



Other Shell Variables 1/2

String MeaningCDPATH=dirs Directories searched by cd ; allows

shortcuts in changing directories ; un-set by default

HOME=dir Home directory ; set by login (from/etc/passwd file)

IFS=’chars’ Input field separators ; default isspace, tab, and newline

LANG=dir Directory to use for certain language-dependent programs

MAIL=file Default file in which to receive mail ;set by login


SyntaxVariables



Other Shell Variables 2/2

String MeaningMAILCHECK=n Number of seconds between mail

checks ; default is 600 (10 minutes)PATH=dirlist One or more pathnames, delimited by

colons, in which to search for com-mands to execute

PS1=string Primary prompt string ; default is $PS2=string Secondary prompt (used in multiline

commands) ; default is >SHELL=file Name of default shell (e.g., /bin/sh)TERM=string Terminal type


SyntaxVariables


BaseBoucleFonctionScript

shell-bang # :

#! / b in / sh# comment. . .


SyntaxVariables



! :

i f who | grep roo t > / dev / n u l lthen

echo roo t i s c u r r e n t l y logged onf ii f ! who | grep roo t > / dev / n u l lthen

echo roo t i s not c u r r e n t l y logged onf ii f who | grep roo t > / dev / n u l lthen : # Do noth ing i f pa t t e rn i s foundelse

echo roo t i s not c u r r e n t l y logged onf i


SyntaxVariables



cd pwd kill sleep time

$ pwd/ home / admin$ cd / tmp /$ pwd/ tmp$ k i l l 19524$ date && sleep 2 && dateMon Aug 13 16:30:29 CEST 2007Mon Aug 13 16:30:31 CEST 2007$ t ime ( f i n d / tmp >/dev / n u l l 2>&1)

r e a l 0m0.002 suser 0m0.000 ssys 0m0.004 s


SyntaxVariables



type ulimit umask

$ type mv readmv i s / b in /mvread i s a s h e l l b u i l t i n# Turn o f f w r i t e permiss ion f o r o thers :# Produces f i l e permiss ion −rw−rw−r−−$ umask 002

# Turn o f f a l l permissions f o r group & others :# Produces f i l e permiss ion −rw−−−−−−−$ umask 077

$ man u l i m i t


SyntaxVariables



if

i f [ $counter − l t 10 ]then

number=0$countere lse

number=$counterf i

i f [ ! −d $ d i r ] ; thenmkdir $ d i rchmod 775 $ d i r

f i


SyntaxVariables



for

f i l e _ l i s t =" i n s t a l l _ p e a r l _ h o s t . t x t "HOST_OK= ‘awk ’ / \ [ X \ ] / { p r i n t $2 } ’ $ f i l e _ l i s t ‘HOST_KO= ‘awk ’ / \ [ \ . \ ] / { p r i n t $2 } ’ $ f i l e _ l i s techo " host OK"f o r host i n $HOST_OK; do

echo $hostcheck_tcp −H $host − t 1 −p 10001

doneecho " host KO"f o r host i n $HOST_KO; do

echo $hostcheck_tcp −H $host − t 1 −p 10001

done


SyntaxVariables



while

whi le [ − f t o t o ] ; doecho Waits leep 1

done


SyntaxVariables



continue

f o r i i n ‘ seq 7 ‘ ; doi f [ $ i −eq 5 ] ; then

cont inuef iecho $ i

done123467


SyntaxVariables



break

f o r i i n ‘ seq 7 ‘ ; doi f [ $ i −eq 5 ] ; then

breakf iecho $ i

done1234


SyntaxVariables



until

u n t i l [ − f t o t o ] ; doecho Waits leep 1

done


SyntaxVariables



case

case $1 i n # Match the f i r s t argno | yes ) response = 1 ; ;−[ tT ] ) t ab l e =TRUE ; ;∗ ) echo " unknown opt ion " ; e x i t 1 ; ;

esac


SyntaxVariables



name()

count ( ) {l s | wc − l

}count13


SyntaxVariables



alias

a l i a s l l = ’ l s − l a t r ’a l i a s l s = ’ l s −−co lo r =auto ’a l i a s omega= ’ cd ~/ p r o j e c t s / omega ’


SyntaxVariables



exit

i f [ $# −eq 0 ]then

echo " Usage : $0 [−c ] [−d ] f i l e ( s ) " 1>&2e x i t 1 # Er ro r s ta tus

f i


SyntaxVariables



getopts

whi le getopts " hzgc : " Option ; docase $Option i n

h ) usage ; e x i t 0 ; ;c ) COPY=1 ; HOST=$OPTARG ; ;g ) GENERATE=1 ; ;∗ ) echo " Unimplemented op t ion chosen . "e x i t 3 ; ;

esacdone

s h i f t ‘ expr $OPTIND − 1 ‘


SyntaxVariables



echo

user@myhost :~$ echo $LC_ALLen_US .UTF−8user@myhost :~$ echo "$LC_ALL"en_US .UTF−8user@myhost :~$ echo ’$LC_ALL ’$LC_ALL


SyntaxVariables



export et eval

$ command=echo$ expor t command$ parameters =" d i f f e r e n t parameters "$ expor t parameters$ eval $command $parametersd i f f e r e n t parameters


SyntaxVariables



nohup

nohup - run a command immune to hangups, with output to anon-tty

$ nohup l snohup : appending output to ‘ nohup . out ’$ cat nohup . outssh−GKxbwc9627ssh−kXMoX18526ssh−tEcuR24939


SyntaxVariables



read

$ read f i r s t l a s t addressSarah Caldwel l 123 Main S t ree t$ echo " $ las t , $ f i r s t \ n$address "Caldwel l , Sarah123 Main S t ree t


SyntaxVariables



return

Exit the function with status n or with the exit status of thepreviously executed command.

r e t u r n


SyntaxVariables



set

set −vx # Read each command l i n e ;# show i t ; execute i t ;# show i t again# ( w i th arguments )

set +x # Stop command t r a c i n gset −o noclobber # Prevent f i l e o v e r w r i t i n gset +o noclobber # Al low f i l e o v e r w r i t i n g

# again


SyntaxVariables



shift

Shift positional arguments (e.g., 2becomes1). If n is given, shiftto the left n places. Used in while loops to iterate throughcommand-line arguments.

$ s h i f t$ s h i f t 2


SyntaxVariables



test - File Conditions 1/2

−b f i l ef i l e e x i s t s and i s a block spec ia l f i l e .

−c f i l ef i l e e x i s t s and i s a charac te r spec ia l f i l e .

−d f i l ef i l e e x i s t s and i s a d i r e c t o r y .

− f f i l ef i l e e x i s t s and i s a regu la r f i l e .

−g f i l ef i l e ex i s t s , and i t s set−group−i d b i t i s se t .

−k f i l ef i l e ex i s t s , and i t s s t i c k y b i t i s set .


SyntaxVariables



test - File Conditions 2/2

−p f i l ef i l e e x i s t s and i s a named pipe ( f i f o ) .

−r f i l ef i l e e x i s t s and i s readable .

−s f i l ef i l e e x i s t s and has a s ize g rea te r than zero .

− t [ n ]The open f i l e d e s c r i p t o r n i s assoc ia ted

−u f i l ef i l e ex i s t s , and i t s set−user−i d b i t i s se t .

−w f i l ef i l e e x i s t s and i s w r i t a b l e .

−x f i l ef i l e e x i s t s and i s executable .


SyntaxVariables



test - String Conditions

s t r i n gs t r i n g i s not n u l l .

−n s1S t r i n g s1 has nonzero leng th .

−z s1S t r i n g s1 has zero leng th .

s1 = s2S t r i ngs s1 and s2 are i d e n t i c a l .

s1 != s2S t r i ngs s1 and s2 are not i d e n t i c a l .


SyntaxVariables



test - Integer Comparisons

n1 −eq n2n1 equals n2 .

n1 −ge n2n1 i s g rea te r than or equal to n2 .

n1 −gt n2n1 i s g rea te r than n2 .

n1 − l e n2n1 i s less than or equal to n2 .

n1 − l t n2n1 i s less than n2 .

n1 −ne n2n1 does not equal n2 .


SyntaxVariables



test - Combined Forms

! c on d i t i o nTrue i f c o nd i t i on i s f a l s e .

cond i t i on1 −a cond i t i on2True i f both cond i t i ons are t rue .

cond i t i on1 −o cond i t i on2True i f e i t h e r c o n d i t i o n i s t r ue .


SyntaxVariables



test - Examples

# While there are arguments . . .wh i le t e s t $# −gt 0# While there are nonempty arguments . . .wh i le [ −n " $1 " ]# I f $count i s less than 1 0 . . .i f [ $count − l t 10 ]# I f the RCS d i r e c t o r y e x i s t s . . .i f [ −d RCS ]# I f the answer i s not y . . .i f [ " $answer " != " y " ]# I f the f i r s t argument i s not a# readable f i l e or a regu la r f i l e . . .i f [ ! −r " $1 " −o ! − f " $1 " ]


SyntaxVariables



trap

Remove a $tmp file when the shell program exits, or if the userlogs out, presses CTRL-C, or does a kill :

t r ap " rm − f $tmp ; e x i t " 0 1 2 15


SyntaxVariables



wait

( / b in / d f / path / n fs ) & cmdpid=$ !t imeout =10(

sleep $t imeoutecho " d f $1 check f a i l e d "( k i l l −9 $cmdpid ) > / dev / n u l l 2>&1

) &watchdogpid=$ !wa i t $cmdpidk i l l $watchdogpid >/dev / n u l l 2>&1


SyntaxVariables



unset

$ echo $ to to

$ t o t o =qsdf$ echo $ to toqsdf$ unset t o t o$ echo $ to to

$


SyntaxVariables



filename

Put filename in script to launch it


SyntaxVariables


AwkSedAutres

Awk

awk ’ { p r i n t $1 } ’ a− f i l eawk ’ { p r i n t $1 " ; " $4 } a− f i l eawk ’ {FS = " ; " } / ^ ’ $ {HOST} ’ / { p r i n t $2 } ’ a− f i l eawk −F ’ | ’ ’ { p r i n t $2 } ’ a− f i l eawk ’ / \ [ $ {PATTERN } \ ] / { p r i n t \ $2 } ’ a− f i l eawk ’ { i f ( $1 == tbsp ) { p r i n t $2 } } ’

tbsp ="$ {TBSP} " a− f i l eawk ’ {FS = " : " } / ^ [ ^ # ∗ ] / { p r i n t $1 } ’ a− f i l eawk ’BEGIN{FS = " : " } $6 ~ / ^ \ / $ /

{ p r i n t " In / : " , $1 } ’ / e tc / passwdawk ’ $3 ~ / : / { p r i n t $1 } ’ a− f i l e


SyntaxVariables


AwkSedAutres

Sed

sed ’ s / ^ / XX / g ’sed − f $ {MEGASED} $ { f i l e }sed " s /#ENV#/ $ { env } / g "sed ’ s / ^KO / / ; s / KO$ / / ’


SyntaxVariables


AwkSedAutres

Autres 1/X

cat << "EOF" | ssh $1 / b in / sh −ps −edf −o comm, args | grep [ h ] t t p d | s o r t −u

$ {ORACLE_HOME} / b in / sq lp l us " / as sysdba " << EOFspool $ {ORACLE_BASE} / admin / $ {ORACLE_SID } / c reate / scorac le . logEOF

cat << EOF | ssh $1 / b in / sh −chown −R $ {ADMCTS_NAME} : $ {ADMCTS_GROUP} $ {HOMEDIR}EOF


SyntaxVariables


AwkSedAutres

Autres 2/X

expect << EOFspawn ssh − t $1 passwd $ {ADMCTS_NAME}expect "New Password : "send " $ {ADMCTS_PASSWD} \ r "expect "Re−enter new Password : "send " $ {ADMCTS_PASSWD} \ r "expect eofEOF


SyntaxVariables


AwkSedAutres

Autres 3/X

cat << "EOF" | ssh $1 / b in / bash −SITES=/ s i t e s

i f [ −d $SITES ] ; thencd $SITESf o r s i t e i n ∗ ; do

NB_PROC= ‘ ps −edf | grep $ s i t e | grep −vc grep ‘i f [ $NB_PROC −eq 0 ] ; then

echo " $ s i t e missing "f i

donef i


SyntaxVariables


AwkSedAutres

Autres 4/X

cat << EOF > $ { SED_FILE }s %172.30.47.11.∗sapome01 .∗# Front−End%172.30.156.142sapome01%s %172.30.47.14.∗sapome04 .∗# Front−End%172.30.156.144sapome04%EOF

cat << EOF | ssh $1 / b in / bash − | tee r e p o r t / $1echo ’uname −a ’uname −aechoEOF


SyntaxVariables


AwkSedAutres

Conclusion

Conclusion


SyntaxVariables


AwkSedAutres

Bibliography

UNIX in a NutshellEd. O’Reilly & Associates


SyntaxVariables


AwkSedAutres

Questions

Questions ?


Technology

Formation sh