Essential Unix

8/6/2019 Essential Unix

1/81

Essential UnixTips, Tricks, and Hints Every DBA Should Know


2/81

2

Topics of Discussion

1. Basic Concepts

2.

Fundamental UnixCommands

3. Working with Multiple Servers

4. Scripts


3/81

3

1. Basic Concepts

Terminology

Case Sensitivity

Unix Building Blocks Wildcards


4/81

4

Terminology

Basic Concepts

Unix Utilities

Unix Kernel

Unix Shell


5/81

5

Unix Utilities

Basic Concepts - Terminology

The collection of various programs that individuals execute fromthe command prompt to perform a variety of tasks

Examples:

listing contents of a directory (ls) Sorting (sort)

Displaying contents of a file (cat, more, head, tail)

Editing files (vi)

Printing (lp)

Searching files for a keyword (grep)

Changing directories (cd)


6/81

6

Unix Kernel


A program that starts running when the system is booted and runscontinuously until the system is shutdown.

Functions of the kernel include file management, disk I/O, processscheduling and management, memory management, systemaccounting, and interrupt and error handling.


7/81

7

Unix Shell


The component that interacts with the user

A program that is executed on behalf of the user upon logon to thesystem and presents the command prompt (typically a $) to the

user Reads input typed into the command prompt, hands off

information to the kernel, and delivers results back to the usersscreen

Environmental settings adjusted via variables

i.e. ORACLE_HOME=/u01/app/oracle


8/81

8

Unix Shell (continued)


env Command to view environment variables

See man page for environ for more variables

Environment variables can be set at login time via .profile a textfile in home directory

$ cat .profile

export ORACLE_HOME=/u01/app/oracle/product/10202

export ORACLE_SID=orclexport LD_LIBRARY_PATH=/u01/app/oracle/product/10202/lib


9/81

9

Unix Building Blocks

Basic Concepts

Pipes

Standard Output

Standard Error Standard Input


10/81

10

Pipes

Basic Concepts Unix Building Blocks

Mechanism that allows the output from one command to be fed asthe input to another

Vertical bar | character is the pipe operator

Example:$ cat oratab | cut f1 d: | sort

dca

drm

dse

dta


11/81

11

Standard Output


a.k.a Standard Out, stdout, filehandle 1

Default location is terminal screen

Redirect stdout to a new file with the redirect operator > Append to an existing file with the concatenation operator >>

Redirect stdout to the input of another command with the pipeoperator |


12/81

12

Standard Output


Examples:

ls l *log

date > rmlogfiles.txt

echo removing log files >>logfile.txt

ls l *.log >>logfile.txt

rm *.log

cat oratab | cut f1 d: | sort 1> sids.txt


13/81

13

Standard Error


a.k.a stderr, filehandle 2

Default location is terminal screen

Redirect stderr to a new file with the redirect operator 2> Append to an existing file with the concatenation operator 2>>

Redirect stderr to the same location as stdout using 2>&1

Pipes typically not used with stderr


14/81

14

Standard Error


Example:

$ cat sids.txt bogusfile 1>stdout.txt 2>stderr.txt

$ cat stdout.txt

dca

drm

dse

dta

$ cat stderr.txtcat: cannot open bogusfile


15/81

15

Standard Input


a.k.a Standard In, stdin, filehandle 0

Default location is keyboard input

Redirect stdin from a new file with the redirect operator


16/81

16

Wildcards

Basic Concepts

A means of matching patterns; usually in file names or whensearching contents of a text file

Sometimes referred to as regular expressions

Examples:

*An asterisk matches any number of letters or digits

? Question mark matches a single letter or digit

[ ] Square brackets matches range of letters or digits


17/81

17

Wildcard Examples (cont)

Basic Concepts

list all temp tablespace data files that contain two alphanumeric characters

after the word temp in the name. i.e. temp01.dbf, tempAB.dbf

ls temp??.dbf

list all files that start with a letter followed by a digit followed by any

combination of letters or characters and ends with .log; i.e. a2test.log

ls [a-z][0-9]*.log


18/81

18


1. Basic Concepts

2. Fundamental Unix Commands


4. Scripts


19/81

19


Commands That Provide Information

Commands That Do Something


20/81

2

0

Commands that Provide Information

Fundamental Unix Commands

Online Documentation (man command)

Examining Contents of Files

File and Directory Information

System Information


21/81

2

1

man command

Fundamental Unix Commands that provide information

Syntax: man command_name or man k command_name

-k option used for keyword searches Optional arguments specified within [ ]

Multiple arguments followed by

Read entire man page towards the end is list of bugs, relatedcommands, related files, exit codes, or other information


22/81

22

Examining Contents of Files


head

tail

more

wc

view

grep


23/81

2

3

head

Fundamental Unix Commands that provide informationExamining Contents of Files

Displays only first few lines of files. Useful in scripts to extractspecified number of lines

Default is 10 lines

Syntax: head [-number] [file] Examples:

head listener.log

head 24 listener1.log listener2.log

X=`cat output.txt | head -1`


24/81

2

4

tail


Syntax: tail [+-number] [-f] [file]

+number start relative to beginning of file

-number start relative to end of file

-f keeps file open and continuously updates; cancel with

Examples:

tail -f listener.log

cat output.txt | tail +30

tail 24 alert_orcl.log


25/81

25

more


Displays text file one page at a time

Use space bar to advance

Type q to quit

Type h for help on more options

Syntax: more [file]

Examples:

more listener.log

cat listener.log | more


26/81

26

wc


Word Count Counts lines, words, and chars

Syntax: wc [ -lwc ] file

Default is all

Example: (shows how many trace files)

ls l *.trc | wc -l


27/81

27

view


Read-only version of vi editor

Same as vi R command

Syntax: view file

Type :q to exit


28/81

28

grep


Global Regular Expression Print

Finds patterns within a file

Syntax: grep [-cilnv] pattern file

-c returns count of lines matching pattern

-i ignores case

-l only prints names of files with matches

-n precede each matching line with its line #

-v prints all lines except those matching pattern


29/81

29

grep - Examples


grep c ORA-1650 alert_orcl.log

1

grep i ^ora-1650 alert_orcl.log

ORA-1650: unable to extend rollback segment R01 by 128 intablespace RBS

grep l ORA alert*.log

alert_orcl.log

alert_orcl2.logalert_orcl.old.log

ps ef | grep v oracle

Shows all non-oracle owned processes


30/81

30

File and Directory Information


ls

find

du

fuser


31/81

31

ls - List directory contents

Fundamental Unix Commands that provide informationFile and Directory Information

Syntax: ls [-aAdlqrR1] [file]

-a lists all files including those beginning with .

-A same as a minus directories . and ..

-d if argument is a directory, only list its name

-l long format including owner, group, size, time

-q prints non-printable characters as a ?

-r reverse order sort

-R recursively list subdirectories and their contents-1 prints only one entry per line


32/81

32

find


Recursively descends directory tree to find files

Syntax: find path options

path directory to begin searching

Options:

-print prints file names found

-ls prints long listing of file names found

-mtime n prints file names modified n days ago

-mtime +n prints file names modified >n days ago-mtime n prints file names modified


33/81

33

find


Options (continued)

-name pattern prints file names matching pattern

-perm mode prints file names matching permissions-size +nc prints file names >n bytes

-size nc prints file names


34/81

34

find Examples:


Show files greater than 2 GB

find /u02/oradata size +200000000c ls

Show files less than 2 GB

find /u02/oradata size -200000000c ls

Find files modified in the last 24 hours

find /u01/app/oracle type f mtime 1 -ls

Remove trace files older than 30 daysfind /u01/app/oracle/admin name *.trc mtime +30

\

exec rm {} \;


35/81

35

find Examples (continued)


Find scripts where new users are created

find $ORACLE_HOME type f name *.sql \

exec grep il identified by {} \;

Find scripts that might have passwords

find $ORACLE_HOME type f -exec grep il sqlplus \;

Find files owned by oracle that are world readable

find / -user oracle perm o+r ls Find files owned by oracle that are world writeable

find / -user oracle perm o+w ls


36/81

36

find Examples (continued)


Find linked files

find $ORACLE_HOME type l -ls

Compress export files that are larger than 1MB and older than 30 days

find $ORACLE_HOME name *.dmp size +1048576c \

-mtime +30 exec compress {} \;


37/81

37

du Disk Usage


Syntax: du [-sk] [file]

Options:

-s sum of all files and subdirectories

-k results in K bytes rather than blocks

Example:

cd /u02/oradata

du sk * | sort -nr


38/81

38

fuser


Identifies processes using a file

Use just before deleting a file

Syntax: fuser [-u] [file]

-u shows username too

Example: (note, 9i and above use and datafiles)

fuser u user_data.dbf

user_data.dbf: 708o(oracle)

ps ef| grep 708oracle 708 1 0 Jul 26 ? 35:11 ora_smon_drm


39/81

39

System Information


ps

crontab

w

who

ipcs


40/81

40

ps List info about processes

Fundamental Unix Commands that provide informationSystem Information

Syntax: ps [-ef] [-o options]

Options*:

-e list info about every process

-f full listing

-o info on specific attributes like mem, cpu

*lots more options available, consult man page


41/81

41

ps Examples


Show %CPU, Userid, ProcessID, and command

ps eo pcpu,user,pid,comm

Show Memory, %Memory, Userid, Process IDand command

ps eo vsz,pmem,user,pid,comm | sort nr | head -5


42/81

42

crontab Scheduled Job execution


To list crontab file

crontab l To edit crontab file

crontab e

See man page for formatMust have environment variable EDITOR set,

typically, vi


43/81

43

System uptime


Four similar commands:

uptime shows how long system is up w shows uptime and load average

who b shows date/time of last reboot

whodo l same as w command


44/81

44

ipcs inter process communication status


Displays info on shared memory segments and

semaphores Syntax: ipcs a

Difficult to determine which instance segmentsbelong to

See also, Oracles sysresv command


45/81

45

Commands that Do Something

ipcrm

[un]compress

touch script

split

mknod

nohup

*these commands discussed in white paper

Fundamental Unix Commands

tar*

ln*

vi*

kill*

umask*

dos2unix / unix2dos*

rm*

sort*

uniq*

cut*


46/81

46

ipcrm Removes ipc resources

Syntax: ipcrm [-m shmid] [-s semid]

Options:-m shared memory segment to remove

-s semaphore id to remove

Note, see also Oracles sysresv command

Fundamental Unix Commands that do Something


47/81

47

compress / uncompress Compresses or uncompresses a file

Syntax:

compress fileuncompress file

Compressed files have .Z naming extension

Example:compress myexport.dmp

ls

myexport.dmp.Z



48/81

48

touch Changes a files access and modification time

Creates an empty file if name does not exist

Useful in scripts to create flag files

Syntax:

touch filename



49/81

49

script Records an interactive terminal session

Useful for documenting series of typed

commands Use to stop recording

Syntax:

script [-a] scriptfilename-a appends to scriptfilename



50/81

50

split

Breaks up a large file into pieces

Syntax:

split -linecount filenamesplit b nm filename

-linecount subfiles of linecount lines

-b nm subfiles of n MB in size

Examples:split 1000 fullexport.dmp

split b 50m coldbackup.tar



51/81

51

mknod

Make a special file

Typically used to create a pipe

Syntax:mknod filename p

Example:

mknod /tmp/mypipe puncompress /tmp/mypipe&

imp file=/tmp/mypipe



52/81

52

nohup No hang up Used when running a process in the

background to prevent it from dying if the

parent is killed Output sent to file nohup.out unless redirected

Example:

nohup ten_hour_imp.ksh &exit # log off box and go home.



53/81

53


1. Basic Concepts



4. Scripts


54/81

54


Old school vs SecureSchool

ssh

scp

rsync


55/81

55

ssh Secure Shell

De-facto standard for remote access client

Offers encryption for remote login Prevents clear text passwords on network

Use in place of telnet, rlogin, rsh, ftp, rdist

Working With Multiple Servers


56/81

56

Remote commandsWorking With Multiple Servers

Non-secure

(old school)

Secure

(preferred)

rsh sshtelnet ssh

ftp sftp

rcp scprlogin Slogin

Rdist rsync


57/81

57

scp Remote CopyAllows files to be copied to other servers

without id/password

Especially useful prior to 8i when copying archlogs to standby servers

Syntax: scp [host:]srcfile [host:]destfile

Examples:scp oraprod01:/var/opt/oracle/oratab ./oratab.1 # gets

file

scp oratab.1 oraprod02:/tmp/oratab.2 # puts

file



58/81

58

ssh secure remote Shell Connects to remote host and executes the

specified command without id/password

Useful for running command across multiplehosts

Syntax:

ssh host command

Example:ssh oraprod01 cat /var/opt/oracle/oratab



59/81

59

rsync Remote Distribution Open source utility for keeping files in sync

across multiple servers

Secure alternative to rdist

Useful for disaster recovery scenarios

Useful for porting scripts, config files, etc.

Typically installed by system administrators

Many options; man page is worth while

More info google, wiki, etc



60/81

60


1. Basic Concepts

2

. Fundamental Unix Commands


4. Scripts


61/81

61

4. Putting It All Together With Scripts

Scripting Techniques

Sample Scripts


62/81

62

Scripting Techniques

Use Command Line Arguments

Use Variables Use `Backticks`

Derive Values, Dont Hardcode

Validate Use of Script

Putting It All TogetherWith Scripts


63/81

63

Use Command Line Arguments

Write Scripts with flexibility in mind

Pass values in via command line, donthardcode

Shell built in variables:

$# - number of arguments on command line

$0 the name of the shell script$1 the first argument passed in

$2 - $n the 2nd through nth arguments

Putting It All TogetherWith Scripts Scripting Techniques


64/81

64

Use Variables

Using variables makes codeeasier to maintain

Avoid hard coding values!



65/81

65

Use `Backticks` Backticks (tilde ~ key) allow the output from a

shell command to be assigned to a variable

For Example:NOW=`date`

MYDIR=`pwd`

NUM9iSIDS=`grep c 9.2 $ORATAB`



66/81

66

Derive Values, Dont Hardcode

Instead of hard coding values, derive themfrom config files

Most common is ORACLE_HOME, can bederived given the SID (passed in?)

Example:

ORACLE_HOME=`grep ^$SID: $ORATAB | cut d: -f2`

Note, $ORATAB can usually be derived based onoutput from uname command:

i.e. Linux=/etc/oratab, Solaris=/var/opt/oracle/oratab



67/81

67

Validate Use of Script If a script should only be executed by

the oracle user, then validate it!

if [ `whoami` != oracle ]; then

echo error. Must be oracle

exit 1

fi



68/81

68

Validate Use of Script

If a script should (or should not) be executedon a given host, then validate it!

THISHOST=`uname a | awk {print $1}`

if [ $THISHOST = $PRODHOST ]; then

echo Danger! Dont run inproduction

exit 1

fi


P tti It All T th With S i t


69/81

69

Sample Scripts


resourcepig

dbsize

dbcount runsql

Finding default passwords (white paper only)

Finding users passwords (white paper only)



70/81

70

Resource pig


$ cat pig

# pig -- list top 5 cpu & mem users

echo "%CPU USERID PID COMMAND"

ps -eo pcpu,user,pid,comm | sort -nr | head -5

echo "MEM %MEM USERID PID COMMAND"

ps -eo vsz,pmem,user,pid,comm |sort -nr |head -5



71/81

71

dbsizePutting It All TogetherWith Scripts

#!/opt/bin/perl

#

# assumes OFA directory layout

$argc = @ARGV;if( $argc != 1 )

{

print STDERR "$0: Wrong number of arguments.\n";

print STDERR "Usage: $0 sid\n";

exit 1;}

$sid=$ARGV[0];

$sid =~ tr/A-Z/a-z/;

P tting It All Together With Scripts


72/81

72

dbsize (continued)Putting It All TogetherWith Scripts

@Lines=`du -sk /u*/oradata/$sid /u01/app/oracle/admin/$sid`;

$total = 0;

foreach $line (@Lines) {($size) = split('\s',$line);

$total = $total + $size;

}

$total = $total / 1024;

print "$total MB\n";

Putting It All Together With Scripts


73/81

73

dbcountPutting It All TogetherWith Scripts

#!/bin/ksh

# dbcount - show entries in oratabs across all servers

cat /u01/app/oracle/share/etc/dbhosts | while read LINE

do

case $LINE in\#*) ;; # ignore comment-line

*) HOST=`echo $LINE | awk '{print $1}' -`

echo "********* ${HOST} **************

# note n option redirects input to /dev/null

ssh -n $HOST cat /var/opt/oracle/oratab |

grep -v "^#" | grep -v "^*"

;;

esac

done

Putting It All Together With Scripts


74/81

74

runsqlPutting It All TogetherWith Scripts

#!/opt/bin/perl

# loops through the oratab file and for each instance,

# logs on as ops$oracle via sqlplus and executes runsql.sql

# with the output being sent to runsql.txt.

open (oratab,"


75/81

75

runsql (continued)Putting It All TogetherWith Scripts

($sid,$home,$yn) = split ":"; # parse out the fields

if ( $yn eq 'Y' ) {

$ENV{"ORACLE_SID"} = $sid;

$ENV{"ORACLE_HOME"} = $home;

$ENV{"PATH"} = $home."/bin:/bin:/usr/bin";

$ENV{"LD_LIBRARY_PATH"} = $home."/lib:/usr/openwin/lib";

print STDERR "processing $sid\n";

system("sqlplus -silent \/ \@runsql.sql >>runsql.txt");}

} # while oratab

close (oratab);

F th R di


76/81

76

Further Reading

Books

Journals

Web www.unixguide.net/unixguide.sh

tml

F th R di B k


77/81

77

Further Reading Books

Just about anything from O'Reilly &Associates (www.ora.com) is good.

Personal favorites: Practical Unix & Internet Security, 3rd Edition

Unix for Oracle DBAs, Pocket Reference

Mastering Regular Expressions, 2nd Edition

Oracle Scripts

Perl for Oracle DBAs Oracle and Open Source

sed & awk, 2nd Edition

Learning the vi Editor, 6th Edition

Learning Perl, 3rd Edition


78/81

78

Further Reading Journals

http://dir.yahoo.com/Computers_and_Internet/s

oftware/operating_systems/unix/magazines/


79/81

79

Further Reading Web

http://www.unixguide.net/unixguide.shtml (A personal favorite. Greatreference)

http://www.unixguide.net

http://isc.faqs.org/faqs/unix-faq/faq/

http://www.ugu.com/ http://www.perl.org

http://www.ee.surrey.ac.uk/Teaching/Unix/

http://www.sans.org/

http://www.cert.org

http://www.orafaq.org http://www.sun.com/bigadmin/


80/81

80

Q & A


81/81

Thank you!

Essential Unix for DBAs

Kristopher Cook

[email protected]

Documents

Essential Unix