View
0
Download
0
Category
Preview:
Citation preview
Pipes and Filters
Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License
See http://software-carpentry.org/license.html for more information.
The Unix Shell
Pipes and Filters Introduction
shell
Pipes and Filters Introduction
shell pwd mkdir cd nano ls rm . rmdir .. mv
cp
Pipes and Filters Introduction
shell
More powerful
when combined
pwd mkdir cd nano ls rm . rmdir .. mv
cp
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$ wc *.pdb
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$ wc *.pdb
* is a wild card
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$ wc *.pdb
* is a wild card matches zero or more characters
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$ wc *.pdb
* is a wild card matches zero or more characters so *.pdb matches all filenames ending in .pdb
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$ wc *.pdb
wc cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb pr
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$ wc *.pdb
word count
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$ wc *.pdb
word count counts lines, words, and characters in files
Pipes and Filters Introduction
$ ls molecules
cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb
$ cd molecules
$ wc *.pdb
20 156 1158 cubane.pdb
12 84 622 ethane.pdb
9 57 422 methane.pdb
30 246 1828 octane.pdb
21 165 1226 pentane.pdb
15 111 825 propane.pdb
107 819 6081 total
$
Pipes and Filters Introduction
$ wc -l *.pdb
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
$
report only lines
Pipes and Filters Introduction
$ wc -l *.pdb
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
$
report only lines use -w for words or -c for characters
Pipes and Filters Introduction
Which file is shortest? 20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
Pipes and Filters Introduction
Which file is shortest?
Easy to see when there are six…
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
Pipes and Filters Introduction
Which file is shortest?
Easy to see when there are six…
…but what if there were 6000?
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$
redirect output to a file
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$
redirect output to a file create file if it doesn't exist
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$
redirect output to a file create file if it doesn't exist overwrite it if it does
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$
no screen output
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$ ls lengths
lengths
$
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$ ls lengths
lengths
$ cat lengths
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
$
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$ ls lengths
lengths
$ cat lengths
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
$
concatenate files
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
$ ls lengths
lengths
$ cat lengths
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
$
concatenate files in this case, only one so file contents printed to screen
Pipes and Filters Introduction
$ sort lengths
9 methane.pdb
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
107 total
$
Pipes and Filters Introduction
$ sort lengths > sorted-lengths
$
Pipes and Filters Introduction
$ sort lengths > sorted-lengths
$ head -1 sorted-lengths
9 methane.pdb
$
Pipes and Filters Introduction
$ sort lengths > sorted-lengths
$ head -1 sorted-lengths
9 methane.pdb
$
get the first line of the file
Pipes and Filters Introduction
$ sort lengths > sorted-lengths
$ head -1 sorted-lengths
9 methane.pdb
$
get the first line of the file this must be the PDB file with the fewest lines, since sorted-lengths holds files and line counts in order from least to greatest
Pipes and Filters Introduction
$ sort lengths > sorted-lengths
$ head -1 sorted-lengths
9 methane.pdb
$
get the first line of the file this must be the PDB file with the fewest lines, since sorted-lengths holds files and line counts in order from least to greatest
not particularly obvious
Pipes and Filters Introduction
$ sort lengths | head -1
9 methane.pdb
$
Pipes and Filters Introduction
$ sort lengths | head -1
9 methane.pdb
$
a pipe
Pipes and Filters Introduction
$ sort lengths | head -1
9 methane.pdb
$
a pipe use output of left side
Pipes and Filters Introduction
$ sort lengths | head -1
9 methane.pdb
$
a pipe use output of left side as input to right side
Pipes and Filters Introduction
$ sort lengths | head -1
9 methane.pdb
$
a pipe use output of left side as input to right side without creating temporary file
Pipes and Filters Introduction
$ wc -l *.pdb | sort | head -1
9 methane.pdb
$
don't need to create lengths file
Pipes and Filters Introduction
$ wc -l *.pdb | sort | head -1
9 methane.pdb
$
This simple idea is why Unix has been so successful
Pipes and Filters Introduction
$ wc -l *.pdb | sort | head -1
9 methane.pdb
$
This simple idea is why Unix has been so successful Create simple tools that:
Pipes and Filters Introduction
$ wc -l *.pdb | sort | head -1
9 methane.pdb
$
This simple idea is why Unix has been so successful Create simple tools that: – do one job well
Pipes and Filters Introduction
$ wc -l *.pdb | sort | head -1
9 methane.pdb
$
This simple idea is why Unix has been so successful Create simple tools that: – do one job well – work well with each other
Pipes and Filters Introduction
$ wc -l *.pdb | sort | head -1
9 methane.pdb
$
This simple idea is why Unix has been so successful Create simple tools that: – do one job well – work well with each other 10 tools can be combined in 100 ways
Pipes and Filters Introduction
running program
Pipes and Filters Introduction
running program process
Pipes and Filters Introduction
standard input
Pipes and Filters Introduction
standard input stdin
Pipes and Filters Introduction
standard output stdout
Pipes and Filters Introduction
shell
Pipes and Filters Introduction
$ wc -l *.pdb > lengths
Pipes and Filters Introduction
wc
$ wc -l *.pdb > lengths
Pipes and Filters Introduction
wc
$ wc -l *.pdb > lengths
Pipes and Filters Introduction
wc
$ wc –l *.pdb > lengths
lengths
Pipes and Filters Introduction
wc
$ wc –l *.pdb | sort
sort
Pipes and Filters Introduction
wc
$ wc –l *.pdb | sort | head -1
sort head -1
Pipes and Filters Introduction
This programming model called pipes and filters
Pipes and Filters Introduction
This programming model called pipes and filters
A filter transforms a stream of input into a
stream of output
Pipes and Filters Introduction
This programming model called pipes and filters
A filter transforms a stream of input into a
stream of output
A pipe connects two filters
Pipes and Filters Introduction
This programming model called pipes and filters
A filter transforms a stream of input into a
stream of output
A pipe connects two filters
Any program that reads lines of text from standard
input, and writes lines of text to standard output,
can work with every other
Pipes and Filters Introduction
This programming model called pipes and filters
A filter transforms a stream of input into a
stream of output
A pipe connects two filters
Any program that reads lines of text from standard
input, and writes lines of text to standard output,
can work with every other
You can (and should) write such programs
Pipes and Filters Introduction
pwd mkdir cd nano ls rm . rmdir .. mv
cp
Pipes and Filters Introduction
pwd mkdir wc cd nano sort ls rm head . rmdir .. mv
cp
Pipes and Filters Introduction
pwd mkdir wc cd nano sort ls rm head . rmdir tail .. mv split
cp cut uniq
Pipes and Filters Introduction
pwd mkdir wc * cd nano sort > ls rm head | . rmdir tail .. mv split
cp cut uniq
Pipes and Filters Introduction
pwd mkdir wc * cd nano sort > ls rm head | . rmdir tail < .. mv split ?
cp cut uniq
August 2010
created by
Greg Wilson
Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License
See http://software-carpentry.org/license.html for more information.
Recommended