Unix Basics

Presented by: Patton FastUser Support Grouphelp-line: help@msi.umn.edu 612-626-0802personal: pfast@msi.umn.edu 612-625-6573

Research.We make it happen.

I. Warnings!

II. Basic Concepts

III. Basic Commands

IV. The Fun Stuff

V. Resources

Please Ask Questions!

Outline

• If you delete a file, it is GONE! The file can only be retrieved from a backup.

• If you overwrite a file it has been changed forever. The original file can only be restored from a backup.

• Home directories are backed up nightly.

• The scratch spaces are NOT backed up!

Bottom line? Be Careful With Your Files!

Warnings!

Scratch Space Time Limits!

• Regatta (/regscratch1 and /regscratch2) – 7 days, 50 Gb

• SP (/scratch1 and /scratch2) – 7 days, 50 Gb

• Altix (/scratch1 and /scratch2) – 14 days

• Netfinity (/scratch1 and /scratch2) – 14 days

• SDVL (/scratch) – 14 days

• BSCL (/wrk and /wrk2) – 14 days

Bottom line? Be Careful With Your Files!

Warnings!– scratch space

Netfinity Linux Cluster160 Xeon processors224 Gb of memory2.0 Tb of storage

www.msi.umn.edu/user_support

Basic Concepts– resources

IBM SP370 Power3 processors663 Gb of memory

4 Tb of storage

http://www.msi.umn.edu/sp http://www.msi.umn.edu/netfinity

SGI Altix176 Itanium2 processors352 Gb of memory

5 Tb of storage

http://www.msi.umn.edu/altixhttp://www.msi.umn.edu/pseries

IBM pSeries312 Power4 processors664 Gb of memory

7 Tb of storage

Computational Genetics Lab16 CPU SunFire 68004 Dell PCs1 Mac G51 SGI Fuel1 SGI Tezro1 IBM IntelliStation

Basic Sciences Computing Lab48 CPU SGI Altix16 CPU SGI Onyx49 SGI Octanes8 CPU Sun UltraSparcIII1 SunBlade 20002.5 Tb of storage

www.msi.umn.edu/user_support

Scientific Dev. & Vis. Lab9 SunBlade 2000s 7 SGI Octanes1 SGI Onyx

Basic Concepts– resources

Med. Chem./Supercomputing Inst. Vis. Workstation Lab

3 SGI Octanes3 SGI O2’s 2 SunBlade 2000s

http://www.msi.umn.edu/bscl http://www.msi.umn.edu/vwl

http://www.msi.umn.edu/sdvlhttp://www.msi.umn.edu/cgl

Basic Concepts– console

Console Login:

• Enter username

• Enter password

Consoles normally have a graphical-windows type interface, and always have priority over remote logins.

Console Logout:

Logout procedures vary between consoles. Make sure you logout! Otherwise your files and account are open to the next person. You could be logged out by an administrator.

After your first login you should change your initial password to something that is easy for you to remember, but not easy for someone to guess.

%> yppasswd (Institute machines)

%> passwd (general)

You should NEVER give your password to anyone!

If you forget your password, send a message to help@msi.umn.edu and your password will be reset.

Note: Never share accounts or passwords!

Basic Concepts– changing passwords

ssh [options] machine secure shell; a program for logging into a remote machine providing encrypted communications between machines

options:-l login name-X sets environment variables for porting X-display

example:

type: ssh -l pfast sp open a secure connection for the user pfast on the machine sp.msi.umn.edu

type: ssh bi2 open a secure connection to bi2.msi.umn.edu with current username

SSH clients:

http://www.msi.umn.edu/user_support/ssh

Basic Concepts– remote login

Why?

• So that you get all of your email sent to one account.

• Makes it easier for sys admins to communicate with you.

How?

1) Create the file .forward in your home directory

2) On the first line:

your.email@desired.location

3) Create this same file in all of your accounts that you want to forward.

Basic Concepts– forwarding email

man [options] command displays manual pages about command-k keyword search for a command

info command displays Info document of command

When in doubt, check the man/info pages!

Need extra details? Check the man/info pages!

Basic Concepts– command line help

Basic Concepts– file structure overview

/

bin@ home/ lib@ scratch/ usr/

user1/ user2/

file1

mail/ source/ work/

proj1/ proj2/

.cshrc

input output

bin/ lib/ local/

user1/ user2/

dir1/ dir2/

file1/ file2/directory ( / )executable ( * )link ( @ )

more*

Basic Concepts– metacharacters

? single character wild card

* wild card, arbitrary number of characters

~ home directory of current user

~name home directory of user name

. current directory

.. parent directory (back one)

command [options] argument

example:

ls -al /scratch

command name

[options] (usually preceded by a dash)

argument (quite often a filename or directory)

Note: UNIX commands, options, and arguments are all CasE seNsiTive!

Basic Concepts– command anatomy

Command Description

ls [options] dirname list the contents of dirname

pwd display full pathname

mkdir dirname create the directory dirname

cd dirname change to the directory dirname

mv [options] source destination move a file or directory

cp [options] source destination copy a file or directory

rm [options] source remove a file or directory

ssh [options] hostname remote login

scp [options] user@host1:file user@host2:file remote copy; file transfer

Basic Commands

skip basic commands

ls [options] dirname list the contents of dirname

options:

-a list all files including hidden files[hidden files are preceded by a ‘.’; eg .cshrc]

-l long listing showing ownership, permissions, and links

examples:

type: ls /home/user/temp view the contents of a directory with absolute pathname /home/user/temp

type: ls ../../temp list the contents of a directory using a relative path.

Listing Contents

mkdir dirname create the directory dirname

examples:

type: mkdir work create the directory work/ in the current working directory

type: mkdir work/proj1 create the directory proj1/ in the work/ directory

type: mkdir /wrk/user2 create the directory user2/ in the /wrk/ directory

Make Directory

cd change to your home directory

cd dirname change to the directory dirname

examples:

type: cd ~tom change to tom’s home directory

type: cd /wrk/user2 change to the directory /wrk/user2

Change Directory

mv [options] presname newname rename a file

mv [options] srcfile destdir move a file to another directory

options:

-i confirm overwrites

example:

type: mv ~user1/file ./outputfile moves file from user1’s home directory to the current working directory and renames it output file

Note: Be careful when overwriting files!

Moving or Renaming

cp [options] srcfile destfile copy a file to another filename

cp [options] srcfile destdir copy a file to another directory

options:

-i confirm overwrites-R recursively copy a directory and its

contents, copies symbolic links

example:

type: cp -R proj1 proj2 copy the directory proj1/ and name it proj2/

Note: Be careful when overwriting files!

Copying

cp [options] srcfile destfile copy a file to another filename

cp [options] srcfile destdir copy a file to another directory

options:

-i confirm overwrites-R recursively copy a directory and its

contents, copies symbolic links

example:

type: cp -R proj1 proj2 copy the directory proj1/ and name it proj2/

Note: Be careful when overwriting files!

Copying

Tip:If you have large files to move from the scratch directories,copy (cp) them rather than moving (mv) them.

rm [options] filename remove a file

options:

-f remove without prompting-i confirm removal-r recursively remove a directory and its contents

example:

type: rm -rf /scratch/user2/dir1 remove the directory /scratch2/user2/dir1and its contents

Note: Be careful when removing files!Especially with wild cards (*,?)!

Removing

scp [options] host:file1 host:file2 secure copy; copy files between hosts on a network using the ssh protocol

options:-r recursive (used for directories)

example:type: scp pfast@wind:”output/*.out” . copy the output files from host wind

with username pfast to current directory

type: scp -r mail origin:mail_copy copy the mail directory to the origin and rename the directory mail_copy

Note: You can use sftp if scp is not available.

Remote Copy

scp [options] host:file1 host:file2 secure copy; copy files between hosts on a network using the ssh protocol

options:-r recursive (used for directories)

example:type: scp pfast@wind:”output/*.out” . copy the output files from host wind

with username pfast to current directory

type: scp -r mail origin:mail_copy copy the mail directory to the origin and rename the directory mail_copy

Note: You can use sftp if scp is not available.

Remote Copy

Tip:If you have large files to transfer, use sftp instead ofscp.

chmod [options] who ops permission filename change file permissionsuse ls -l to view permissions

options

–R recursively change permissions

who can be any combination of:u (user) g (group)o (other/world) a (all or ugo)

ops adds or takes away permission, and can be:+ (add permission) – (remove permission)

permission can be any combination of:r (read) w (write) x (execute) X (check user first)

Changing Permissions

history display list of most recent commands

!string repeat command beginning with string

up/down arrows scroll up and down through most recent commands; csh and tcsh

left/right arrows use for editing current line

Control-E end of lineControl-A beginning of line

examples:

type: !ls repeat last ls command

Command Repetition

TAB finish the current command, filename, or or directory if possible, or show theControl-D possible completers

(works with c and tc shells)

Command Completion

tar [options] files collect or extract files or directories into or from an archive

options:c create a new archivex extract files from archivef store files in archivev print function letter and name of files– store to standard output

example:type: tar xvf file.tar extract the files and or directories from the

archive, file.tar

Tarring

compress file compress file and rename it file.Z(the program automatically renames the file)

uncompress file.Z uncompress file.Z and rename it file

gzip file compress file and rename it file.gz(the program automatically renames the file)

gunzip file.gz uncompress file.gz and rename it file

Compression

Process Control

Control-C cancel a foreground job

Control-Z stop (interrupt) a foreground job

jobs list of background jobs

bg run stopped job in the background

fg run stopped job in the foreground

& appended to the end of a command will place that job in the background

example:type: blastp swissprot test.seq > test.out & run WU-BLAST job in the

background

ps [options] display the status of the current processes and the process id-number

options:-a all processes-u display processes owned by a particular user

jobs shows any jobs that are currently running in the background or suspended

Process Control

kill –9 id-number terminate a process owned by you; id-number (process id) can be found with the ps command

or

kill %job-number terminate a process owned by you; job-number can be found with the jobscommand

Process Control

du -sk file display total disk usage of file or directory in kilobytes

df display the current disk usage for all connected file systems

stat file view attributes of a file or directory

quota -v list your disk quota

Other Commands

cat [options] files dumps the contents of files to the screen

more [options] files a pager, that lets you view one page at a time

less file similar to more, but better

Viewers

Graphical:

nedit filename if not available, ask to have it installed

xemacs filename (jot no longer works on SGI)

Command-line:

pico filename full-screen text editor available on most UNIX systems

vi filename full-screen text editor available on virtually every UNIX system

Editors

The Fun Stuff– redirect

> command output redirection (create new, overwrites existing file)

>> command output redirection (append)

< command input redirection (from file)

examples:

how to: combine individual sequence files into one file

type: cat seq1 seq2 > seq

how to: email a file to a collaborator

type: mail -s “results” collab@umn.edu < new_blast_results

The Fun Stuff– piping

command1 | command2 | … directs standard output of one command into standard input for the next command

examples:

type: ls -al | more look at the ls output one page at a time

type: qstat -a | grep user1 check the queue on the origin for any processes being run by user1

grep [options] string file find and print out lines containing the string in a file or multiple files

options:

-i ignore case

examples:

how to: extract sequence information from protein structure files

type: grep SEQRES pdbfile | cut -c20-70 > seqs

The Fun Stuff– pattern searching

.cshrc shell startup file

Some things to add to your .cshrc file:

alias alias-string com-string abbreviate a command string with an alias string

examples:

type: alias la ls -a create a shortcut for ls -a

type: alias q qstat -u pfast check if you have any jobs running in the queue (origin)

module load(unload) software load or unload the environment settings for a particular software

source /usr/local/gcg/gcgstartup runs the gcgstartup script to set the necessary env. variables

The Fun Stuff– .cshrc file

UNIX in a Nutshell by O’Reilly

Developing Bioinformatics Computer Skills by O’Reilly

http://www.iats.missouri.edu/servlets/knowledgebase/article/422

http://www.geek-girl.com/unix.html

http://riceinfo.rice.edu/Computer/Documents/Unix/unix1.01.pdf

Resources