Introduction to Linux Command Line
Introduction to Linux Command Line
September 13, 2016
Patrick Bills,Research Consultant
Institute for Cyber-Enabled Research
Introduction to Linux Command Line
Goal : gain understanding of:● What is Linux & Unix● Know how to get connected to a linux server● The unix 'shell': Using a computer with only
the keyboard● Commands and Utilities● Manipulating Files● Scripting
Getting Started
Introduction to Linux Command Line
How we will work and help you
During exercises we want to know who needs help, and when to move on
Use the green and red sticky notes provided to help me help you.
No sticky => I am workingGreen => I am done and ready to move onRed => I am stuck and need help
After we move on, or start new exercise, remove Green sticky
Introduction to Linux Command Line
Class structure
Get ConnectedTerminal and Shell conceptsWorking with Directories and FilesEnvironment : configuration, settings and variablesEditing files Working with a data fileScripting with shellUsing other scripting environments (Python, Perl, C)Processes (& , fg, bg, ps aux, top)GUI X11 (gedit)
Introduction to Linux Command Line
MSU HPC runs Linux
We'll use the MSU High Performance Computing Cluster for this course.
● consistent experience (Windows, Mac laptop diversity)
● it's a system we know● highly availability● course is pre-requisite
for other HPCC courses
Introduction to Linux Command Line
Get Started
1. Install/identify laptop software to connect2. Connect to a Unix machine with secure shell3. Does your account work? troubleshoot if necessary4. test basic commands
Step 1: Install Software (if you haven't)Windows : install "MobaXterm" free Unix emulation program
When installed, start program and then start New Session
Mac OS X : includes a unix terminal, but install Xquartz for GUIOpen the application /Applications/Utilities/Terminal.app
Introduction to Linux Command Line
Get Connected
Mac or Linux: Terminal
Start Terminal from /Applications/UtilitiesType ssh [email protected]
Enter password - it's hidden and doesn't look like you are typing
Windows:
Start Moba XtermStart new session
Select SSH Enter host = hpcc.msu.eduUser = your netidPort = 22 Click 'connect'
Introduction to Linux Command Line
What are shell commands?
The shell lets you run programs both utilities and programs you write, and capture output of those programs● Shell can be the glue script to bind programs on the system together● Shell language is one tool in your Unix automation toolbox ● Other tools: SED, awk, grep, Perl, Python, Ruby, C, etc● These all make it easy to ask the system to do something.● Unix command can be called from the above scripting languages, and
those scripts can be called from the shell● Unix philosophy : best tool for the job, but allow the tools to work
together
Introduction to Linux Command Line
What is a command?$<command> <argument(s)> <optional file name>
ls = list files ; arguments -l = long format; -a = all files including ‘hidden’ filesA command is a verb. The the thing that takes action. The command is a program that runs.The options are the ‘adverbs’ - qualifiers of those command ‘verbs.’• wc -m dictionary.txt• we count, characterly, the dictionary text• ls -l $HOME• we list, longly, our home’s files
System information: date command$ date$ date -h$ date --help
Using Shell Commands and Utilities to interact with Linux
Introduction to Linux Command Line
Exploring files and folders with 'ls'
Step one, explore your home directory with 'ls' Note when giving command examples, $ at the beginning is the prompt. Please try these commands
$ ls shows files
$ ls -l shows files in long view
$ ls -al shows all files ; what new things do you see?
$ ls /mnt/home lists folders of current users
$ ls /mnt/home/userid enter your own user id here
Now try someone else's folder - what happens?
$ ls /mnt/home/somebody
Introduction to Linux Command Line
About Files and Folders
Linux has single directory 'tree', separated by slash, the top is the 'root' All additional disks are connected on /mnt, 'mounted' No concept of drive letters
Introduction to Linux Command Line
On our system /mnt/home/ are all user home directories folders/mnt/scratch/ are working files/bin and /usr/bin and /opt/software software
1. Use pwd to determine your current point in the tree 2. Use ls -l to see all files in your folder; note it starts with two entries
single dot (. ) short cut representing current folder double dot ( .. ) short cuts representing parent folder
3. cd = change directory. cd .. => go up one4. ~ is shortcut for your home directory5. Files are specified by the "path" through directory tree6. Path can be full ( /mnt/home/billspat/.bashrc ) starting with /7. or relative , from the current location ~/.bashrc
Exploring files and folders
Introduction to Linux Command Line
File Permissions
Linux has method to keep files private and save. permission for user, user groups, and all others. drwxr-xr-x 4 cliff user 1024 Jun 18 09:40 hello.txt
-rw-r--r-- 1 cliff user 767392 Jun 6 14:28 backup.tar.gz
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^
| | | | | | | | | | |
| | | | | owner group size date time name
| | | | number of links to file or directory contents
| | | permissions for world
| | permissions for members of group
| permissions for owner of file: r=read, w=write, x=execute
type of file: - = normal file, d=directory, l=symbolic link, etc
Compare ls -l ~ and also ls -l /mnt/scratch/billspat/class
Introduction to Linux Command Line
File Permissions continued
User, Group, others have Read, Write or Execute permissions
Read and Write make sense, but execute? Programs must have this to run
Permission are set with the "chmod" command, and ownership set with "chown" See the man pages or
You can only change these for files you are the owner of
Don't grant other/world permission on your home folder. Keep it private.
Introduction to Linux Command Line
Linux includes a built in manual for nearly all commands. Type 'man' followed by the command e.g $man rm
To navigate the man pages use the arrow keys to scroll up and down or use the enter key to advance a line, space bar to advance a page, letter 'u' to go back a page. Use the q key to quit out of the display.The manual pages often include these sections:
● Name--a one line description of what it does.● Synopsis--basic syntax for the command line.● Description--describes the program's functionalities.● Options--lists command line options available for this program.● Examples--examples of some of the options available● See Also--list of related commands.
Note that options can be with single dash "-" or double dash "--"
Learning commands with manual pages
Introduction to Linux Command Line
review the man pages of common file/directory commands
ls mkdir cp pwd cat
● How can you list all files in a folder, including hidden files that start with a dot (.) ?
● Which command would create a folder "class" in your home directory? Can you do that? what are the options for this command?
● What does pwd do?
● The 'cd' command changes directory, but what happens when you examine the manual page with man cd ? Why?
Learning about commands with 'man'
Introduction to Linux Command Line
External Resources to learn Linux and Bash
http://ss64.com/bash/ "An A-Z Index of the Bash command line for Linux." with links to 'man' manual page for each. It includes many commands that you will never use, or are specific to certain versions of linuxhttps://fosswire.com/post/2007/08/unixlinux-command-cheat-sheet/ useful cheatsheet organized by topichttps://cvw.cac.cornell.edu/linux/
Full intro to Linuxhttp://guide.bash.academy
step by step intro with lots of explanatory texthttp://www.grymoire.com/Unix/Awk.html
complex intro to awk
Introduction to Linux Command Line
Specifying Files and Folders Wildcards
Exercise: copy files for this class are in /mnt/scratch/billspat/class to your own class folder in your home directory Which folders are there? How would you copy my 'docs' folder in my class folder to yours?
Method 1: create doc folder in your home, and copy each filemkdir ~/class/doc
cd ~/class/doc
cp /mnt/scratch/billspat/class/doc/bash_cheat_sheet.pdf .
Method 2: use short cutmkdir ~/class/doccp /mnt/scratch/billspat/class/doc/* ~/class/doc/*
Method 3: copy ALL folders all at oncecp -R /mnt/scratch/billspat/class/* ~/class/
Introduction to Linux Command Line
Shell Environment Variables
Shell maintains, and you can set 'variables' that the shell uses for configuration and in your scripts. Variables start with $, and can be seen with echo $VARNAME● Explore common variables with the echo command and list what they
are$HOME $USER $SHELL $PATH
● Combine variables in an echo command to make a greeting. ● Set a variable of your own, then use in the sentenceGREETING=Hello; echo $GREETING, $USER - welcome to $HOME
● Create a shortcut using the "alias" builtin, which has the basic syntaxalias shortcut="command" Note: must surround command in quotes
Introduction to Linux Command Line
$PATH, one of most important variables
BASH looks for programs in each folder listed in $PATH
When you type a program, how does the shell find it? Let's try the which command, it shows the location of programs...
which date # in /bin
which less # in /usr/bin
which matlab # in /opt/software
each of these folders in the path variable.
Introduction to Linux Command Line
Command input/output
Most commands take text as input and output results as textCommon ways to get text in and out
$ ls -l output text to the screen$ ls -l > filelist.txt redirect text into a new file$ ls -l | grep txt pipe text into another command (chaining)
$ ls $HOME data or file as command line option$ wc -l < filelist.txt redirect file as input into command$ cat filelist.txt | wc -l output file, pipe into commandWhy did we have to use 'cat' in the above command ?
Introduction to Linux Command Line
Utilities for Data
wc count words, lines or characters
example: who | wc -l # number of users logged in
grep finds patterns in files or data, returns just matching lines
example: who | grep $USER # ARE YOU LOGGED IN?
awk can extract individual columns from data, among many other things. awk <options> '{ <program> }' filename
example: awk -F ',' '{print $3}' data.csv
using the field separator option, show column 2 of file with commas
uniq reduce list to uniq values
example: who | awk '{print $1}' | uniq
Introduction to Linux Command Line
Utilities for data
sort given a list of items, sort in various waysexample: who | sort
head list only top n lines of filetail list only last n lines of filezip create zip of several filesunzip unzip tar create (tar -c ) or extract (tar -x) "tape archive" filetr transpose characters example: tr ':' ',' < /etc/passwd
Exercise : using the man pages, find out what the default number of lines that head and tail will display, and how to limit those to just one line
Introduction to Linux Command Line
Exercise : find largest file by sorting
Challenge : find the largest file in a folder, for example /mnt/research/common-data/Bio/genomes/hg19
How do you show all files with sizes? Which column is the size in, each column separated by one or more spaces
sort command has a -k option specifies which column to sort on. You want to sort it numerically, and man sort tells the option needed to "compare according to numerical value" See man sort
Can you combine these commands with the pipe |
Introduction to Linux Command Line
Working with data files
Assure you've copied all file/folders into ~/class folder● Can you unzip the file in class/data folder? see man unzip● Challenge : what is the name of the data file in the zip?● how long is the file (how many rows?)● how many columns● can you show just the column names?● can you show only rows for Michigan?
Commands to use : wc , grep
Bonus: use grep and awk to show poll numbers for Obama
Introduction to Linux Command Line
Data file challenge
Answerscd ~/class/data
unzip data.zip
ls
wc -l polls.csv
head -1 polls.csv
head -1 polls.csv
Bonus
cat polls.csv | awk -F ',' '{print $2}'
Introduction to Linux Command Line
Using Nano ; a simple text editor
When you must edit a file directly on the system, need a terminal-only editor, e.g. nano
Start: nano myfile.txtExit: ^x (control + x), then answer 'y' , then press enter
Example command: ^k cut, ^u uncut (paste)
Introduction to Linux Command Line
Edit a file with nano
Confirm you can open the file greet.sh in class/code folder
method 1 nano ~/class/code/greet.sh
method 2cd ~/class/code
nano greet.sh
This is an example shell script. you can execute it by specifying the path and the file ~/class/code/greet.sh
but first must make it "executable" with chmod u+x ~/class/code/greet.sh
#!/bin/bashGREETING=HelloQ=`df -h $HOME | tail -n 1 | awk '{print $4}'`
echo $GREETING, $USER! Your Home dir is $Q full.echo It\'s currently `date +%c`
Introduction to Linux Command Line
Modify shell script
There is another script in the code folder written in python
Exerise: view this program to see what it doesWhat does it do? How do you think you run it?
python ~/class/code/forecast.py 48824
Exercise: add this command to the greet.sh script and see if it runs. Pick any zipcode
Bonus : use shell arguments ( $1 ) instead of hard coded zip code
Introduction to Linux Command Line
Process
Unix is a series of processes
ps list your processps aux list all processestop list user process using the most CPU
press ctrl+c to exit top
run process in the backgroundpython ~/class/code/forecast.py 48824&
ps
Introduction to Linux Command Line
Using GUI
Windows: Moba XTerm works out of the boxMac: must launch Xquartz, then ssh -X [email protected] in the xterm
demo
Launch GUI editor$ gedit bonus try $ gedit&
Open the shell script and editBonus : Launch matlab
Introduction to Linux Command Line
Discussion
Q/A and optional work
Exercise : can you compile the class folder into a zip file to save it?
How will you use Linux?
Compiler C programs?