CISC3130: Redirection & Pipe

1

Xiaolan ZhangSpring 2011

CISC3130: Redirection & Pipe

2

AgendaContinue of last class: file systemMisc topics

Some productivity tipsMisc. commandsFile manipulation utilities

Input/output redirection and pipePutting things together

3

Last classHierarchical file structureAbsolute pathname & relative pathname

/home/staff/zhangzhang, staff/zhang, ../../etc/passwd

/

home

staff

bin

zhang

etc

passwd

dev

cdrom tty24

lib

A tree of directories and files

“..”: refers to parent dir“.”: current directory“/”: root and seperatorin file names“~”: home directory

4

Long listingTo get more information about each

file[zhang@storm Demo]$ ls -altotal 32drwxr-xr-x 5 zhang staff 4096 2008-01-16 16:01 .drwxr-xr-x 41 zhang staff 4096 2008-01-16 16:01 ..drwxr-xr-x 2 zhang staff 4096 2008-01-16 15:55 CCodes-rw-r--r-- 1 zhang staff 38 2008-01-16 16:01 .HiddenFile-rw-r--r-- 1 zhang staff 53 2008-01-16 15:57 READMEdrwxr-xr-x 2 zhang staff 4096 2008-01-16 15:55 SampleCodes

drwxr-xr-x 4 zhang staff 4096 2008-01-16 15:56 ShellScriptes

Total disc space taken in blocks (1024 Byte)

d means directory

Who has permission to read/write the file User name of the owner and its group

5

Long listing explainedField 1

1st Character – File Type: First character specifies the type of the file.- normal file, d directory, s socket file, l link file

Next 9 characters – File PermissionsField 2 – Number of links

Second field specifies the number of links for that file

Field 3 – Owner: specifies owner of the file

Field 4 – Group: specifies the group of the file

Field 5 – Size: specifies the size of file. Field 6 – Last modified date & time:

date and time of last modification of the file

Field 7 – File name

6

File permissionsEach file is associated with permission info.

Differentiate three type of users: owner user, users from same group as owner, others

Three type of accessRead (r): use “cat” to open a file to read, use “ls” to list

files/directories under a directoryWrite (w): modify the contents of the file,

create/remove files from the directoryExecute (x): execute the file, or “cd” or “search” the

directory for file

Trying to list other’s directory[zhang@storm ~]$ ls ../roche/ls: cannot open directory ../roche/: Permission

denied

7

AgendaContinue of last class: file systemMisc. topics



8

Misc. CommandsSend a file to the printer:

lpr <fileName>The file should be of format that the printer

recognizes, e.g., text file, postscript file (.ps)!which: show the full path name of a command

$ which bash/bin/bash

How does shell find a command ? Environment variable PATH stores a list of paths

to search for programs: “set | grep PATH” or “echo $PATH”, “set” to show all variable settings

PATH=$PATH:$HOME/bin:.Built-in commands: history, set, echo, etc.

9

Some useful tips Bash stores the commands history

Use UP/DOWN arrow to browse themUse “history” to show past commands

Repeat a previous command!<command_no>

e.g., !239 “!<any prefix of previous command>

E.g., !g++

Search for a commandType Ctrl-r, and then a stringBash will search previous commands for a match

File name autocompletion: “tab” key

10

Customize your shell environmentTo change your login shell

[zhang@storm ~]$ chshChanging shell for zhang.Password:New shell [/bin/bash]: /bin/bashShell not changed.

Bash configuration files(in home dir, file name start with dot).bash_logout: A list of commands to be executed when

leave shell .bash_profile: a list of commands to be executed when log

in.bashrc: a list of commands to be executed when open a

new shell.bash_history: a list of commands you have been typing

11

Environment VariablesSetting environment variables

PATH=$HOME/bin:$PATH

Environment variable PATH stores a list of paths that bash searches for programsTo see the current settings of an environment

variable:set | grep PATH

echo $PATH //set to show all variable settingsPATH=$PATH:$HOME/bin:.PS1="You rang? " export PATH PS1

Can also set and export in one line: export PS1="Yes? “

12

Set environment variablesSetting environment variables

PATH=$HOME/bin:$PATH

Environment variable PATH stores a list of paths that bash searches for programsTo see the current settings of an environment

variable:set | grep PATH

echo $PATH //set to show all variable settingsPATH=$PATH:$HOME/bin:.PS1="You rang? " export PATH PS1

Can also set and export in one line: export PS1="Yes? “

13

Create customized command shorthand Aliases

alias ls='ls –F’alias rm=‘rm –i’ ## so that you have to

confirm the removal

14


Access online manualSome productivity tipsMisc. commandsFile manipulation utilities


15

What’s in a file ? So far, we learnt that files are organized in a

hierarchical directory structureEach file has a name, resides under a directory,

is associated with some admin info (permission, owner)

Contents of file:Text (ASCII) file (such as your C/C++ source

code)Executable file (commands)A link to other files, …

To check the type of file: “file <filename>”

16

Some useful file related utilities

Counting # of lines, words and characters in fileswc [option]…[FILE]….wc [OPTION]… --file0-from=F // to be discussed later

To search files for lines that match a patterngrep [OPTIONS] PATTERN [FILE…]

Examplesgrep “global warming” articles-v option: lines that don’t match the patternWhere did I define/access a variable named

gNumOfOperations ?grep gNumOfOperations *.[ch]

If no file is specified, search standard input for given patternFor redirection/pipeline

17

Sort commandSort lines of text file sort [OPTION]… [FILE]…Many options to control sorting order

-r: reverse the normal order-n: sort in numeric order

By default, the command sort in alphabetic order-nr: sort in reverse numeric order+n: sort starting at n+1-th field

18

Compare file contentsSuppose you carefully maintain diff.

versions of your projects (so that you can undo some changes), and want to check what’s the difference.cmp file1 file2: finds the first place where two

files differ (in terms of line and character)diff file1 file2: reports all lines that are

different

19




20

• On startup, each program has three “files” created/opened for it• Standard input, by default is linked to

keyboard• Standard output, by default is linked to

terminal window• Standard error, by default linked to

terminal window

Standard input, output, error

0 1

2

21

Simple exampleA very simple C program

#include <stdio.h>main() { char yourName[256];

printf ("Your name ?\n"); if (fgets (yourName,256,stdin)==NULL) fprintf (stderr,"No input"); else printf("hello, %s\n", yourName); }

22

Input/Output RedirectionOn command line, one can redirect these three filesTo redirect standard output to a disk file:

$ command [ [ - ] option (s) ] [ option argument (s) ] [ command argument (s) ] > FILENAME

Execute the command, sending its standard output to specified fileExisting content of the file is deleted

For example:ls –lt > InfoFilelist.txt

To append standard output to a file: use >> instead of > grep “tax reform” *.txt > outputgrep “fuel efficiency” *.txt >> output

23

Input/Output Redirection (cont’d)To redirect standard error to a file, instead

of keyboard$ command [ [ - ] option (s) ] [ option

argument (s) ] [ command argument (s) ] 2> ERRORMSGS

Examples:[zhang@storm ~]$ ls abcls: cannot access abc: No such file or directory[zhang@storm ~]$ ls abc 2> error[zhang@storm ~]$ more errorls: cannot access abc: No such file or directory

24

User > and 2> togetherTo split error messages from normal output

[zhang@storm ~]$ ls research.tex abcls: cannot access abc: No such file or directoryresearch.tex[zhang@storm ~]$ ls research.tex abc 2> error

> output[zhang@storm ~]$ cat errorls: cannot access abc: No such file or directory[zhang@storm ~]$ cat outputresearch.tex[zhang@storm ~]$

This is useful for running a command that might take long time to finish, or generates very long output …

25

More on redirectionTo redirect both output and error to same file:

./a.out < tt > dd 2> dd : does not work. Error output is not captured.

sort file.txt > dd 2>&12>&1: redirect error output to same place as standard

outputgrep numOfStudents 2>dd >&2

>&2: redirect standard output to same place as error output

To discard output, redirect it to /dev/null/dev/null: a special virtual file, “a black hole”./a.out > /dev/null 2>&1I don’t want to see the output or error message,

nor do I want them saved to a file …

26

Input/Output Redirection (cont’d)To read standard input from a file, instead

of keyboard$ command [ [ - ] option (s) ] [ option

argument (s) ] [ command argument (s) ] < FILENAME

Examplesmail zhang –s “Question” < proj1.cpp./a.out < values.txt //a.out is your program that reads integers

from standard input and calculate the sum

Environment VariablesUse command env to show all environment

variables and their values

27

28

Combining commands togetherHow many files are there under current

directory ?ls > tmpwc –l < tmprm tmp

Sort current online user by alphabetic order

Is some user login to the system now ? (using grep)

Is file “tmp” listed ?

29

Pipe: getting rid of temporary filePipe: connect the output of one program to

the input of another programAny prog. that reads from standard input

can read from pipe, similarly for the standard outputwho am i | ./a.out | wcThe individual program/command knows

nothing about redirection and pipe

30

Rule of compositionPipe: one of the fundamental contributions

of UNIX systemDesign programs to be connected with

other programsRead/write simple, textual, stream-oriented

formatsRead from standard input and write to standard

outputFilter: program that takes a simple text

stream on input and process it into another simple text stream on output

31

Command Pipeline: how ?Pipe

an inter-process communication mechanism provided by kernel

Has a reading end and a writing endAny data write to writing end can be read back

from the reading endRead/write pipe is no different from

read/write files

Reading endWriting end

32

Command Pipeline: how ?Shell set things up

create a pipe, “start” two programs simultaneously, with their input/output redirected to the reading/ending end of pipe

33

The workings of shellFor each command line, shell creates a new

child process to run the commandSequential commands: e.g. date; who

Two commands are run in sequencePipelined commands: e.g. ls –l | wc

Two programs are load/execute simultaneously

Shell waits for the completion, and then display prompt to get next command …

34

Important concept: ProcessEarly computers run a job from starting to

endMultiprogramming was popularized later

To load multiple programs in memory and switch between them when one is waiting for I/O => increase CPU utilization

Timesharing: a variant of multiprogramming, in which each user has an online terminal (multiple users sharing the system)

35

ProcessA process is an instance of a running program

It’s associated with a unique number, process-id.OS stores its running state

A process is different from a programwc, ls, a.out, … are programs, i.e., executable files

which program […]When you run a program, you start a process to

execute the program’s codeMultiple processes can run same program

At any time, there are multiple processes in the system One of them is running, the rest is either waiting for I/O,

or waiting to be scheduled

36

Access to Instructions

The hard disk is too slow to provide instructions to CPU. So programs are first loaded into main memory, which is much faster.The CPU can then access the instructions more quickly.

37

Loading Program Programs are stored in secondary storage (hard

disks, CD-ROM, DVD) To process data, CPU requires a working area, the

Main Memory Also called: RAM (random access memory),

primary storage, and internal memory. Before a program is run, it must first be copied

from the slow secondary storage into fast main memory Provides the CPU with fast access to instructions

to execute.

38

ps commandTo report a snapshot of current

processes: psBy default: report processes belonging to

current user and associated with same terminal as invoker.

Example:[zhang@storm ~]$ ps PID TTY TIME CMD15002 pts/2 00:00:00 bash15535 pts/2 00:00:00 ps

List all processes: ps -e

39

BSD style output of psLearn more about the command, using man ps

[zhang@storm ~]$ ps axuUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1 0.0 0.0 2112 672 ? Ss Jan17 0:11 init [3]

root 2 0.0 0.0 0 0 ? S< Jan17 0:00 [kthreadd]

root 3 0.0 0.0 0 0 ? S< Jan17 0:00 [migration/0]

root 4 0.0 0.0 0 0 ? S< Jan17 0:00 [ksoftirqd/0]

root 5 0.0 0.0 0 0 ? S< Jan17 0:00 [watchdog/0]


root 7 0.0 0.0 0 0 ? S< Jan17 0:00 [ksoftirqd/1]

root 8 0.0 0.0 0 0 ? S< Jan17 0:00 [watchdog/1]


40

Run program in backgroundTo start some time-consuming job, and go

on to do something else$ command [ [ - ] option (s) ] [ option

argument (s) ] [ command argument (s) ] &wc ch * > wc.out &Shell starts a process to run the command, and

does not wait for its completion, i.e., it goes back to reads and parses next command

Shell builtin command: waitKill a process: kill <processid>

41

Some useful commandsTo let process keep running even after you log

off (no hangup)nohup COMMAND &Output will be saved in nohup.out

To run your program with low prioritynice [OPTION] [COMMAND [ARG]...]

42

Some useful commandsTo start programs at specified time (e.g.

midnight)at [-V] [-q queue] [-f file] [-mldv] timespec...By default, read programs from standard input:[zhang@storm assignment]$ dateMon Jan 31 21:51:38 EST 2011[zhang@storm assignment]$ at 10pm < todojob 15 at Mon Jan 31 22:00:00 2011[zhang@storm assignment]$ more todoecho "HI!"ls | wc –l > temp

43

The Power of PipeWho is using the most CPU ?

ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

Find out how many subdirectories are there ?

Display the content of last edited file (under current directory)… cat `ls –t | head -1`

44

SummaryWe have looked at some basic “start up”

topics.Get around in the file systemSome basic files related commandsIntroduction to shell

Redirection and pipeProcess concept

Documents

CISC3130: Redirection & Pipe