Introduction to UNIX / Linux - 7

Summer 2015SILICON VALLEY UNIVERSITY

CONFIDENTIAL 1

Introduction to UNIX / Linux - 7

Dr. Jerry Shiao, Silicon Valley University

SILICON VALLEY UNIVERSITY CONFIDENTIAL 2Summer 2015

Introduction UNIX/Linux Course File Sharing

On a Computer System, collaboration on Software Project need to share files and directores related to the project.

File Sharing Methods Duplicate (Copies) of Common Files. Common Login for Team members. Common Group for separate Team member Logins. File Sharing Via Links.

Hard Links Soft / Symbolic Links

Redirection and Piping Standard files, stdin, stdout, stderr. Redirection of standard files. Pipes in UNIX : Pipes and Filters perform complicated tasks that cannot

be performed with a single command. Combining Pipes and Redirection operators. C shell and error redirection. FIFOs: Unrelated or independent process communication.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 3

Copyright @2005 Pearson Addison-Wesley

Introduction UNIX/Linux Course

File Sharing Duplicate Shared Files

Problem: Working of file simultaneously. Files are inconsistent.

Team Common Login Problem: Separate personal and team accounts.

High Overhead.Administrator must maintain new accounts per project.

Access Permissions on Shared Files Problem: If User group not restrictive and access cannot always

guarantee to be project members . Team New Group

Problem: User group must ONLY be members of the team. File Sharing Via Links

Link establish connection between file to be shared and directory entries of users who want to have access to this file.

Link provides different access paths to files to be shared. Hard Links: Pointer to the inode of a file. Soft / Symbolic Links: Pointer to the path of a file.




File Sharing Hard Links

Hard link creates a file entry (new file name) in directory with the same inode index as the linked file.

ln [ options ] existing-file new-file ln [ options ] existing-file-list directory - f : Force creation of link. - n : Do not create the link if “new-file” exists. - s : Create symbolic link to “existing-file”. NOTE:

< no “– s” > : Create a hard link to “existing-file”. < “directory” > : The user MUST have execute and

write permission for all the directories in the path leading to the file.




File Sharing Review Inodea) Logical structure of current directory. File, Chapter3, is created in current directory.

b) Contents of current directory. Directory entry contains pair <inode # / filename>.

c) Relationship among a directory entry, Inode Table, and physical file contents.-Inode # index into the Inode Table. -Inode Table entry contains the physical location of file on disk.




File Sharing Hard Links Current Directory

$ ls -iltotal 122630032 -rw-r--r-x 1 sau users 102 2012-10-26 00:47 Chapter12630033 -rw-r--r-- 1 sau users 102 2012-10-26 00:47 Chapter22630034 -rw-r--r-- 1 sau users 79 2012-10-29 14:23 Chapter3 $ ln Chapter3 Chapter3.hard $ ls -iltotal 162630032 -rw-r--r-x 1 sau users 102 2012-10-26 00:47 Chapter12630033 -rw-r--r-- 1 sau users 102 2012-10-26 00:47 Chapter22630034 -rw-r--r-- 2 sau users 79 2012-10-29 14:23 Chapter32630034 -rw-r--r-- 2 sau users 79 2012-10-29 14:23 Chapter3.hard $ rm Chapter3 $ ls -iltotal 122630032 -rw-r--r-x 1 sau users 102 2012-10-26 00:47 Chapter12630033 -rw-r--r-- 1 sau users 102 2012-10-26 00:47 Chapter22630034 -rw-r--r-- 1 sau users 79 2012-10-29 14:23 Chapter3.hard $

Create hard link. The inode index for linked file and file are the same. Link count is 2.

Remove only removes the entry in the directory. File is physically still on disk. Link count is 1.



Introduction UNIX/Linux Course File Sharing Link Count

Number of different directory entries pointing to inode of the object (i.e. directory or file).

File: Number of hard links to that file. $ ls -il memo6

2630044 -rw-r--r-x 1 sau users 253 2012-10-29 16:16 memo6 Directory: Number of directory entries connecting local

directory to Linux file system. $ ls -a

. .. students students_2 students.gz $ ls -lddrwxr-xr-x 2 student1 student1 4096 2013-10-19 02:06 .$ mkdir test_dir$ ls -a. .. students students_2 students.gz test_dir$ ls -lddrwxr-xr-x 3 student1 student1 4096 2013-10-19 02:06 .




File Sharing Create Hard Linka) Logical structure of current directory. Hard link, Chapter3.hard, is linked to file, Chapter3 in current directory.

b) Contents of current directory. Directory entry contains pair <inode # / filename> for Chapter3.hard. The inode# is same as file Chapter3.

c) Relationship among a directory entry, Inode Table, and physical file contents.-Inode # index into the Inode Table for both Chapter3 and Chapter3.hard.




File Sharing Hard Links Different Directory

$ ls -il memos/memo62630044 -rw-r--r-x 1 sau users 253 2012-10-29 16:16 memos/memo6 $ ln memos/memo6 memo6.hard $ ls -il memos/memo62630044 -rw-r--r-x 2 sau users 253 2012-10-29 16:16 memos/memo6 $ ls -il memo6.hard2630044 -rw-r--r-x 2 sau users 253 2012-10-29 16:16 memo6.hard $ $ rm memos/memo6 $ ls -il memos/memo6ls: cannot access memos/memo6: No such file or directory $ ls -il memo6.hard2630044 -rw-r--r-x 1 sau users 253 2012-10-29 16:16 memo6.hard $






File Sharing Hard Link Different Directory




File Sharing Hard Links Different Directory

$ ls -il memos/memo62630044 -rw-r--r-x 1 sau users 253 2012-10-29 16:16 memos/memo6 $ ln memos/memo6 memo6.hard $ ls -il memos/memo62630044 -rw-r--r-x 2 sau users 253 2012-10-29 16:16 memos/memo6 $ ls -il memo6.hard2630044 -rw-r--r-x 2 sau users 253 2012-10-29 16:16 memo6.hard $ $ rm memos/memo6 $ ls -il memos/memo6ls: cannot access memos/memo6: No such file or directory $ ls -il memo6.hard2630044 -rw-r--r-x 1 sau users 253 2012-10-29 16:16 memo6.hard $





Introduction UNIX/Linux Course File Sharing Hard Links Placed In

Another User Structure /home/user1/class4/demo1 Structure of /home/user2/dir1/demo1

/home/user1$ su…# pwd/home/user1/class4# ln demo1 /home/user2/dir1/demo1# exit$ ls -i demo12630035 demo1

/home/user2$ pwd/home/user2/dir1$ ls -i demo12630035 demo1

1) Creating Hard Link between two users.2) Create Hard Link in /home/user2 pointing to file in /home/user1.3) Target file must have Read/Write permission to Group.



Introduction UNIX/Linux Course File Sharing Hard Links To Another User

User Directory Permissions$ ls -lttotal 40drwx------ 49 student1 student1 4096 2013-10-15 14:44 student1drwx-----x 23 student2 student2 4096 2013-10-15 01:39 student2

$ ls -lt test-rw-rw-r-- 1 student1 student1 99 2013-10-15 15:51 test$ ln test /home/student2/dir1$ ls -lt test-rw-rw-r-- 2 student1 student1 99 2013-10-15 15:51 test

/home/student11) Create Hard Link in /home/student2 /dir pointing to file in /home/student11.2) Target directory, dir1, must have WriteExecute permission to Other.3) Link count for files test is 2.

[student2]$ ls -ld dir1drwxrwx-wx 3 student2 student2 4096 2013-10-15 15:50 dir1[student2]$ ls -lt dir1total 16-rw-rw-r-- 2 student1 student1 99 2013-10-15 15:51 test

/home/student21) Link count for file test is 2.




File Sharing Hard Links Limitations

Cannot use Hard Links between files on different file systems. # ln /usr/bin/zip my_zipln: creating hard link `my_zip' => `/usr/bin/zip': Invalid cross-device link

Moving a hard linked file to different file system causes physical copy of file to be made.

$ ls -il memo6.hard2630045 -rw-r--r-x 2 sau users 253 2012-10-29 16:25 memo6.hard $ ls -il memos/memo62630045 -rw-r--r-x 2 sau users 253 2012-10-29 16:25 memos/memo6 $ mv memo6.hard /tmp $ ls -il /tmp/memo6.hard 953686 -rw-r--r-x 1 sau users 253 2012-10-29 16:25 /tmp/memo6.hard

Editors saves edited file in new files, and remove previous file. The editors, vi and emacs, are safe.




File Sharing Soft Links

Does not have any of the Hard Link issues. Flexible:

Link files across File Systems. Link to directory.

Soft link creates a new file entry (new file name and new inode index) in directory.

ln [ options ] existing-file new-file ln [ options ] existing-file-list directory - f : Force creation of link. - n : Do not create the link if “new-file” exists. - s : Required. Create symbolic link to “existing-file”.




File Sharing Soft Links Current Directory

$ ls -il Chapter32630032 -rw-r--r-x 1 sau users 102 2012-10-26 00:47 Chapter3 $ ln -s Chapter3 Chapter3.soft $ ls -il Chapter3 Chapter3.soft2630032 -rw-r--r-x 1 sau users 102 2012-10-26 00:47 Chapter32630044 lrwxrwxrwx 1 sau users 8 2012-10-29 18:12 Chapter3.soft -> Chapter3 $

Create soft link. 1)The inode index for linked file and file are different. These are different files.2)Original file is file type “-”, an ordinary file. Link file is type “l”, a link file.3)Link count 1. 4)File sizes are different. Link file is 8 bytes, the length of “Chapter3”, the pathname is placed in the link file.




File Sharing Soft Links

Create soft link. 1)The inode index for linked file and file are different. These are different files.2)Original file is file type “-”, an ordinary file. Link file is type “l”, a link file.3)Link count 1. 4)File sizes are different. Link file is 8 bytes, the length of “Chapter3”, the pathname is placed in the link file.




File Sharing Soft Links Placed In Another User

Structure /home/user1/class4/demo1 Structure /home/user2/dir1/demo1

/home/user2$ ls -ld /home/user2drwxr-xr-x 31 user2 users … /home/user2$ ls -ld /home/user2/dir1drwxrwxr-x 2 user2 users … /home/user2/dir1…$ ls -il demo11102983 lrwxrwxrwx 1 user1 users …demo1 -> /home/user1/class4/demo1

/home/user1$ pwd/home/user1/class4$ ln -sf /home/user1/class4/demo1 /home/user2/dir1$ ls -il demo12630047 -rw-rw-r-x 1 user1 users … demo1NOTE: Do not have to be Superuser to create software to another directory. Must have Write/Execute permission at linking directory.

demo1 /home/user1/class4/demo1




File Sharing Soft Links Pros and Cons

Pros Establish symbolic link between files across file systems. Establish symbolic link between directories. Symbolic linked file edited by editor does not change filename.

Cons If the file that the symbolic link points to is moved from one

directory to another, it can no longer be accessed via the link. UNIX has to support an additional file type (the link type) and a

new file has to be created for every link. Symbolic links also result in slow file operations because, for

every reference to the file, the link file has to be opened and read in order to reach the actual file.




Redirection and Piping Standard Files All commands performs one or more of : Input,

Output, Processing Standard files Automatically Opened on Every

Command: Standard Input (stdin) - File Descriptor = 0. Standard Output (stdout) - File Descriptor = 1. Standard Error (stderr) - File Descriptor = 2.

Standard files can be redirected to other files using File Redirection Facilities in UNIX. Connect several commands together to perform a complex

task, that cannot be performed by single command.



Introduction UNIX/Linux Course Redirection and Piping Standard Files: Automatically opened by the Kernel for

every command for te command to read input from and send its output and error messages to.

By default– stdin is associated with keyboard– stdout is associated with display screen– stderr is associated with display screen




Redirection and Piping Input Redirection

command < input-file Input to “command” comes from “input-file” instead of

stdin (i.e. keyboard). cat < tempfile stdin has been has been attached to “tempfile”. cat tempfile stdin still active NOT attached to “tempfile”. “tempfile” is file command line argument to the command.




Redirection and Piping Input Redirection

( keyboard )

( keyboard )

Keyboard detached from stdin and input-file attached.

Input-file

Input-file




Redirection and Piping Output Redirection

command > output-file Send output of “command” to the “output-file” instead

of stdout (i.e. monitor). cat > tempfile stdout has been has been attached to “tempfile”. cat tempfile stdin still active. stdout still active. “tempfile” is file command line argument to the command.




Redirection and Piping Output Redirection




Redirection and Piping Output Redirection Network Environment rsh [ –l <username> ] host [ command ]

Remote shell executes command on host. rsh 192.168.1.47 sort < datafile

ssh [ -l <username> ] host [ command ] Remote login executes command on host. ssh 192.168.1.47 -l root sort < datafile




Redirection and Piping Combine Input and Output Redirection

command < input-file > output-file The “command” takes input from “input-file” instead of

stdin (i.e. keyboard) and sends its output to the “output-file” instead of stdout (i.e. monitor).

cat < lab1 > lab2 stdout has been has been attached to “lab2”. stdin has been attached to “lab1”. The cat command takes its input from the lab1 file and sends its output to the lab2 file.




Redirection and Piping File Descriptor With I/O Redirection

File Descriptor is an integer that the UNIX kernel attaches with every open file to stdin, stdout, and stderr.

Standard Input (stdin) = 0 Standard Output (stdout) = 1 Standard Error (stderr) = 2

File Descriptors can be used for I/O Redirection 0< : Redirect Standard Input 1> : Redirect Standard Output 2> : Redirect Standard Error >& : Redirect one File Descriptor to another File Descriptor

2>&1 : Redirects File Descriptor 2 (stderr) to File Descriptor 1 (stdout). 1>&2 : Redirects File Descriptor 1 (stdout) to File Descriptor 2 (stderr).

Using File Descriptors: “cat > outfile” equivalent to “cat 1> outfile”. “grep John 0< tempfile” equivalent to “grep John < tempfile”.




Redirection and Piping File Descriptor With I/O Redirection

command 2> error-file Error messages generated by “command” sent to stderr are

redirected to “error-file”. $ cat memo 1> letter 2>&1 $ cat lettercat: memo: No such file or directory $ rm letter $ cat memo > letter 2>&1 $ cat lettercat: memo: No such file or directory

Redirects stderr to what stdout was (“letter”) when the argument was encountered.NOTE: If want to redirect stderr, have to use the syntax “2>”.




Redirection and Piping File Descriptor With I/O Redirection Redirect stdout and stderr to the same file by using the

descriptors with > symbol $ cat lab1 lab2 lab3 1> cat.output 2>cat.output

Using string 2>&1 to tell the command shell to make descriptor 2 a duplicate of descriptor 1, resulting in error messages going to the same place the output is goes to $cat lab1 lab2 lab3 1> cat.output.errors 2>&1

Using string 1>&2 to tell the command shell to make descriptor 1 a duplicate of descriptor 2, resulting in output going to the same place as the error messages $ lab1 lab2 lab3 2> cat.output.errors 1>&2

The evaluation for command line content for file redirection is from left to right




Redirection and Piping File Descriptor With Standard Error Redirection command 2> error-file




Redirection and Piping File Descriptor Redirect Stdout and Stderr

cat lab1 lab2 lab3 1> cat.output.errors 2>&1 cat lab1 lab2 lab3 2> cat.output.errors 1>&2




Redirection and Piping File Descriptor Redirect Stdout and Stderr Redirection Evaluated Left to Right

cat lab1 lab2 lab3 2>&1 1> cat.output.errors

Evaluation of command line for file redirection is left to right. One notation is dependent on notation to the left. Evaluation of cat command: 1) Stderr (2) is made a duplicate of stdout (1). 2) Stdout (1) has NOT been redirected. Stderr (2) becomes duplicate of stdout (1) default (console). 3) Stdout (1) is redirected to file “cat.output.errors”. 4) Output cat.output.errors. Errors console.

1 432




Redirection and Piping File Descriptor Redirect Stdout and Stderr Redirection Evaluated Left to Right

cat lab1 lab2 lab3 2>&1 1> cat.output.errors File Descriptor 2, duplicates what File Descriptor 1 is currently

pointing to, stdout. File Descriptor 1, duplicates file cat.output.errors. File Descriptor 2 stdout. File Descriptor 1

cat.output.errors. cat lab1 lab2 lab3 2> cat.output.error 1>&2

File Descriptor 2, duplicates file cat.output.errors. File Descriptor 1, duplicates File Descriptor 2, which is

currently pointing to file cat.output.errors. File Descriptor 2 cat.output.error. File Descriptor 1

cat.output.error.




Redirection and Piping cat lab1 lab2 lab3 2>&1 1> cat.output.errors

(a) and (b)

Stdout and Stderr

does not change.

(c)

Stdout to

“cat.output.errors”.

(d) Command

Semantics




Redirection and Piping File Descriptor With Redirect Stdin, Stdout,

Stderr command 0<input-file 1>output-file 2> error-file Input to ‘command’ comes from ‘input-file’ instead of keyboard,

output of ‘command’ goes to ‘output-file’ instead of the display screen and error messages generated by ‘command’ are sent to ‘error-file’

The file descriptors 0 and 1 are not required as they are the default values

$ sort 0< students 1> students.sorted 2> sort.error $ sort 2>sort.error 0< students 1> students.sorted




Redirection and Piping I/O Redirection Without Overwriting Contents “ > “ Overwrites contents of destination file. “ >> ” Replace “ > “ to Append stdout and stderr at end of

file. “ >> “ default File Descriptor is 1 (stdout). “ 2>> “ Append errors to a file.

ls -l 1>> output.dat 2>> error.log Evaluation: Output appended to output.dat. Error appended to error.log cat memo letter >> stuff 2> error.log Evaluation: Output appended to stuff. Error overwrites error.log




Redirection and Piping I/O Redirection: File Descriptors

[student1]$ exec 4<>test_r4[student1]$ cat >&4Hello test_r4Hello test_r4Hello exit[student1]$ ls -lt test_r4-rw-rw-r-- 1 student1 student1 39 2013-10-17 15:21 test_r4[student1]$ cat test_r4Hello test_r4Hello test_r4Hello exit[student1]$

exec: Execute <command> in place of the current shell (instead of creating a new process).




Redirection and Piping UNIX Pipes ( “ | “ ) Allows stdout of a command to be connected to

stdin of next command. Pipe implemented in main memory. command1 | command2 | … | commandN

Standard output of ‘command1’ is connected to stdin of command2,…,stdout of ‘command N-1’ is connected to stdin of ‘command N’

ls -l | more Using piping. ls -l > temp more < temp No piping. rm temp




Redirection and Piping UNIX Pipes ( “ | “ )

The semantics of a pipe with N commands.

The semantics of the “ls – l | more” command.

The semantics of the “grep ‘John’ < Students | lpr –Pspr” command.




Redirection and Piping UNIX Pipes ( “ | “ )

The semantics of the “ls – l | more” command.

diff datafile - command




Redirection and Piping Redirect stdout to files or another command. Limitation Redirect and Pipe:

Cannot use redirect operators and pipes to redirect stdout of a command to a file and connect it to stdin of another command.

Command1 | tee file1 … fileN | command2 Standard output of “command1” is connected to stdin

of tee, and tee sends its input to file “file1” through “filen” and as stdin of “command2”.




Redirection and Piping Redirect stdout to files or another command.

cat names students | grep “John Doe” | tee file1 file2 | wc –l




C Shell Error Redirection “>&” C Shell Limitation:

File Descriptors cannot be used with ( “<“, “>”, “>>” ). Error Redirection is “>&”, but for both stdout and

stderr. To attach stdout and stderr to different files,

parenthesize the command to have the stdout redirection in the subshell.

% (find ~ -name foo –print > foo.paths) >& error.log

Subshell inherits standard files of the parent shell (stdout and stderr redirected to error.log).

Subshell redirects stdout to foo.paths. Stderr from parent shell still redirected to error.log.

Subshell

Parent Shell executes before SubShell




C Shell stdout and stderr Append “>>&”

command >& file Redirect and append stdout and stderr to “file”.

stdout and stderr Redirect “|&” command1 |& command2 Redirect stdout and stderr of “command1” to

“command2”, i.e., pipe stdout and stderr of “command1” to “command2”.

Pipe and Redirect “|” and “|&” Single Command cat file1 file2 |& grep “John Doe” | sort |& wc –l stdout of grep command (grep BOTH stdin and

stderr) is attached to the stdin of the sort command.




C Shell

Parent Shell( … ) >& error.log

Sub Shell(find ~ -name foo –print > foo.paths) …

(find ~ -name foo –print > foo.paths) >& error.log




C Shell C Shell Special Built-In Variable

set noclobber : Protect from output redirection unset noclobber cat file1 > file2

noclobber set: Error if file2 exists: Creates file2 if not exists. cat file1 >> file2

noclobber set: Error if file2 not exists. file2 exists, append. noclobber clear:

>!, >>!, >>&! Operators override effect of noclobber. cat file1 >! file2 Overwrites existing file2. cat file1 >>! file2 Creates file2, if not exists.




Table I/O and Error Redirection




Table I/O and Error Redirection Bourne Shell Korn Shell C Shell




FIFOs FIFOs (Named Pipes) used for communicating

between two processes on the system. Pipes can only be used for communication between

commands connected via a pipeline. FIFOs created on disk with a name. FIFOs has to

be created and opened before processes. Pipe is in main memory buffer and has no name.

mknod() or mkfifo() to create a FIFO. mkfifo [ option ] file-list Creates FIFOs with pathnames given in “file-list”. - m : Set access permission to “mode” (octal value).




FIFOs $ mkfifo myfifo1 myfifo2 $ ls -lt myfifo1 myfifo2prw-r--r-- 1 sau users 0 2012-11-01 00:26 myfifo1prw-r--r-- 1 sau users 0 2012-11-01 00:26 myfifo2 $ cat myfifo1 | grep "d$" &[1] 25866 $ cat myfifo2 | wc -l &[2] 25888 $ ps -e | tee myfifo1 myfifo2 | wc -l 2 ? 00:00:00 kthreadd 15 ? 00:00:00 kacpid . . .18407 ? 00:00:00 nmbd131131[1]- Done cat myfifo1 | grep "d$"[2]+ Done cat myfifo2 | wc -l

Create FIFOs.File type is “p” for FIFO.

Run command as background process.Block on waiting for FIFO myfifo1.

Run command as background process.Block on waiting for FIFO myfifo2.

Writes to FIFO files, myfifo1 and myfifo2. The blocked commands performs the actions and outputs written to stdout.




FIFOs Semantics for two commands communicate via FIFO. cmd1 < myfifo1 & cmd2 infile | tee myfifo1 | cmd3

Documents

Introduction to UNIX / Linux - 7