COMP 2007: UNIX Operating System

Dr Eleni Mangina

eleni.mangina@ucd.ie

http://www.cs.ucd.ie/staff/emangina

OPERATING SYSTEM

An operating system (OS) is a program that is loaded into the computer’s memory when the system is booted, and it always remains there. It interacts with two agencies:

• Applications (like word-processing software) that run under its auspices

• Users who pass commands to it using a command language interpreter. This too is a program; it translates commands to instructions that can be understood by the operating system.

Why Study Operating System?

• Argument against studying OS

Very few designers/implementers needed

All I need to know is in the manual pages

We are all just going to use Windows’?? anyway

Why Study Operating System?• Arguments in favour

OS is the most used program Need to know about system design in general OS includes all design/implementation problems

• design of efficient interfaces• flexibility, robustness and performance• trade-off between performance and simplicity

Mini-OS in many applications We are not all using Windows-??...

What is an OS?

• Is a program that

manages physical resources acts as an interface between the user and the hardware controls the execution of application programs provides an environment for the execution of programs provides mechanisms and policies to manage the whole

resources of the system controls programs and users interactions

Major Issues in OS• Structure -- how is an operating system organised?

• Sharing -- how are resources shared between users and programs?

• Naming -- how are resources named (by users or programs)?

• Protection -- how is a user/program protected from another?

• Security -- how to restrict the flow information?

• Performance -- why it is slow?

• Reliability & fault tolerance -- when something goes wrong

• Extensibility -- how to add new features?

Major Issues in OS (ctd)• Communication -- how and with whom can we communicate

(exchange information)?

• Concurrency -- how are parallel activities created & controlled?

• Scalability & Growth -- what will happened when demands and/or resources increase?

• Persistence -- how to make data last longer than programs?

• Compatibility -- can we ever do anything new?

• Distribution -- accessing the world of information?• Accounting -- who pays the bills, and how to control resource

usage?

OS Views• An operating system can be seen as:

– a virtual machine (extending the hardware functionality)

– resource manager (managing the hardware resources)

• Virtual Machine– Hide all the low-level details of programming the hardware

– Construct layers of software that provide more and more functionality

– Present a nice and simple view the computer use

OS as Resource Manager• A computer can be seen as a set of resources for

– movement, storage, and processing of data and– control of these functions

• The hardware resources can be– CPU (computation)– Main memory– Secondary memory (disks, tapes, etc)– Network– I/O devices (keyboard, mouse, printers, display, etc)– ...

OS Principles• OS as illusionist

– make hardware resources unlimited– provide illusion of dedicated machine with infinite

memory and infinite processors

• OS as government– protect users from each other– allocate resources fairly and efficiently

• OS as history teacher– learn from the past to predict the future

A brief history of OS• With the earliest computers, the OS was just “code” to which

you linked your program, loaded as a whole into the main memory and ran it– It is just a run-time library

• Simple batch systems were the first real OSs – the OS is stored in the main memory– a single job is loaded (from card reader) into the memory– the control cards of the job tell the OS what to do– run the job (compute, print the outputs, etc.)

• Spooling & buffering allow more than one job to be loaded into tape/disk or main memory

UNIX HISTORY

• 1965: Bell Telephone Laboratories & General Electric Company & Project MAC of MIT new OS Multics

• 1969: primitive version of Multics on a GE 645 & Bell end the participation

• Ken Thopson & Dennis Ritchie design of a file system early version of UNIX

• UNIX Brian Kernighan (Computing Science Research centre)• 1971: Real project: PDP-11, 16K bytes for the system, 8K for

user program, a disk of 512K bytes, limit 64K bytes per file• 1973: Operating system re-written in C 25 installations at Bell

Labs

UNIX HISTORY (cont.)

• AT & T could not market due to a 1956 Consent Decree BUT Universities could use it for educational purposes POPULARITY INCREASED

• 1974: Thompson and Ritchie publication of UNIX description

• 1977: UNIX 500 (125 were universities)• 1977: “ported” to a non-PDP machine• 1984: Developers enhanced UNIX their own way!

variants, 100.000 UNIX system installations!!

WHY UNIX?• The system is written in a high level language• Simpler user interface• Permit complex programs to be built from simpler

programs• Hierarchical file system• Consistent format of files• Simple, consistent interface to peripheral devices• It is a multi-user, multi-process system• It hides the machine architecture from the user

Versions of UNIX

• System V UNIX: AIX, Irix, Solaris, Tru64 Unix, Unicos, UnixWare

• BSD UNIX: BSD/OS, Dynix/ptx, FreeBSD, Mac OS X, NetBSD, OpenBSD, ULTRIX

• UNIX- like systems: Hurd, Linux, Minix, XINU

Multiprogramming• Idea: Expand memory to hold 2 or more programs and switch

among all of them. This process is known as Multitasking or Multiprogramming

• Multiprogramming Systems– keep multiple runnable jobs loaded in memory– overlap I/O operations of a job with computations of another– benefit from I/O devices that can operate asynchronously– require the use of interrupts and DMA– increase the processor utilisation and attempt to optimise

throughput• Central theme of modern operating systems

Timesharing• Idea: is to have multiple users simultaneously using terminals,

with the OS interleaving the execution of each user program in a short quantum of computation

• Timesharing Systems– based on time-slicing– each user feels like using the computer on his/her own– try to optimise response time– allow the users to view, edit, debug, and run their programs

interactively • MIT Multics system (mid-late 60s) was the first large

timesharing system

Real-Time OS

• Idea: OS must guarantee response to physical events in a fixed interval of time

• Real-Time OSs– are used for specialised applications: subway systems, flight

control, factories, nuclear power, etc.– their problem is to schedule all activities in order to meet all

their critical requirements– perform operations at fixed, predetermined times (time

intervals)– allow more user control

• With the use of video applications on modern PCs, all OSs will have real-time requirements

Distributed OS

• Idea: a common operating system shared by a network of independent computers

• Distributed OS– facilitates the use of geographically distributed resources– looks to its users like an ordinary centralised OS– supports communication between parts of a job or

different jobs – supports sharing of distributed resources (hardware and

software)– allows some kind of parallelism, but the speedup is not

an issue

Parallel OS• Idea: a system that can run and manage parallel applications

efficiently on parallel computers

• Parallel OS– support parallel applications composed of several time

consuming subtasks– needs basic primitives for dividing a task into multiple

parallel activities– needs scheduling primitives to map parallel activities into

processors– supports efficient communication between activities– supports synchronisation of activities to co-ordinate

sharing of data

OS Structure

• To understand any OS structure, let’s have a look first at its components and, then, how they are composed and organised

• Note that– it is never simple as it looks like

– the basic concepts exist in some forms in all OSs

– each OS implements them in a slightly different way

– the division between components is not clearly defined

UNIX System Structure

Center: operating system interacts directly with the hardware, providing common services to programs. System kernel: operating system; isolated from user programs. Programs such as shell and editors (ed, vi) interact with kernel with a set of system calls commands!

SCCS: Source Code Control System

Major Components of UNIX:

• Kernel: The master control program of the computer. It resides in the computer’s main memory and it manages the computer’s resources. It is the kernel that handles the switching necessary to provide multitasking

• File System: UNIX organises data into collections calles files. Files may be grouped together into collection called directory files or directories

• Shell: The part of UNIX that interprets user commands and passes them on to the kernel is called the shell. A typical shell provides a command-line interface, where the user can type commands.

• Utilities: A utility is a useful software tool that is included as a standard part of the UNIX operating system. Utilities are often called commands

User Perspective

File system:• Hierarchical structure• Consistent treatment of file data• Ability to create & delete files• Dynamic growth of files• Protection of the file data• Treatment of peripheral devices The file system is organised as a tree with a single root node called

root (written “/”) every non-leaf node of the file system structure is a directory of files and files are either directories, regular files or special device files.

Pathname: how to locate the file in the file system hierarchy Permission: Controlled by access permissions associated with the

file: read, write, execute

File Management• Secondary storage devices are too crude to be used directly

for long-term storage

• A file is – the basic long-term storage entity

– a collection of permanent information that can be read or written

• File system – provides logical objects

– provides logical operations on these objects

– supports directories -- special files that contain other files

File Management• Secondary storage devices are too crude to be used directly

for long-term storage

• A file is – the basic long-term storage entity

– a collection of permanent information that can be read or written

• File system – provides logical objects

– provides logical operations on these objects

– supports directories -- special files that contain other files

File Hierarchy

• File system provides standard file operations– create/delete files– create/delete directories– copy/move files/directories– other basic operations like: read, write, protect, extend,

…

• File system provide also general services– backup– maintaining mapping information– accounting and quotas

Process Management

• An OS executes many kinds of activities– user programs– batch jobs and command scripts– system programs; print spooler, name servers,

file servers, etc

• Each of these “execution entities” is encapsulated in a process

• The OS schedules processes to run

Processes

• A program is just a file on a disk with code that is potentially executable

• A process is one instance of a program in execution. At any time, there may be many processes running copies of a single program (e.g. an editor): each one is a separate and independent process

• A process includes the execution context (program counter, registers, virtual memory, resources, etc.) and all information the activity (program) needs to run

Process Operations• Processes are fundamental OS-provided objects

• The OS operations on processes include:– create a process

– delete a process

– suspend a process

– inter-process communication

– inter-process synchronisation

– create/delete a sub-process

Processing Environment

Program: executable fileProcess: instance of a program in executionexample:

main(argc,argv)int argc;char *argv[];

{/*assume 2 args: source file and target file */if (fork() == 0)

execl(“copy”,”copy”, argv[1],argv[2],0);wait((int *) 0);printf(“copy done\n”);

}

Processing Environment (cont.)

Shell: the command interpreter program that users typicall execute after logging into the system The shell interpretes the first word of a command line as a command name, treating the remaing words as parameters to the commands. It allows three trtype of commands:

• Executable file that contains object code produced by compilation of source code

• Command can be any executable file that contains a sequence of shell command lines

• Internal shell command

example:Who: causes the system to execute the programs stored in the file /bin/who, which prints a list of people who are currently logged into the systemWho &: in the background

Memory Management• The main memory is a direct-access storage device• Programs must be stored in memory to be executed• OS must

– allocate memory to processes (implicitly and explicitly)

– de-allocate it when they are done

– maintain (update) the mapping between virtual and physical memory (page tables)

– Keep track of which parts of memory are in use and which parts are not

– decide how much memory to allocate to each process and when a process should be removed from memory

I/O Management• An important part of the OS kernel is dedicated to I/O

• The OS provides a standard interface between processes (user’s or system’s) and devices

• Device driver – is a process responsible for a given device type

– encapsulate device-specific knowledge, e.g. for device initiation and control, interrupt handling and errors

– there may be a process for each device, or for each I/O request, depending on the OS

Secondary Storage Management• Secondary Storage

– permanent memory– endures system failures (we hope)– block access (Disk), sequential access (tape)

• OS must – provide low-level routines responsible for low-level disk

functions such as disk scheduling, head movement, error handling, etc

– manage the disk space

• The space management can be implemented as low-level routines or handled by the file system

Building Block Primitives

Redirect I/O: processes conventionally have access to three files:• Read from their standard input file (<)• Write to their standard output file (>)• Write error messages to their standard error file (2>)$ ls$ ls > output$ mail mjb < letter$ nroff –mm <doc1> doc1.out 2> errorsPipe: a mechanism that allows a stream of data to be passed

between reader and writer processes$ grep main a.c b.c c.c$ grep main a.c b.c c.c | wc -l

Operating System Services

• Controlling the execution of processes by allowing their creation, termination or suspension and communication

• Scheduling processes fairly for execution on the CPU• Allocating main memory for an executing process• Allocating secondary memory for efficient storage

and retrieval of user data• Allowing processes controlled access to peripheral

devices

System Protection• Protection is a mechanism for controlling the access of

processes or users to the resources of the system

• All resource objects need to be protected– CPU, memory– processes– files– devices, networking

• Protection mechanisms– help to detect errors and

– prevent malicious destruction

Assumptions about Hardware

User mode – Kernel mode

The differences between the two modes are:• Processes in user mode can access their own

instructions and data but not kernel instructions and data processes in kernel mode, however, can access kernel and user addresses

• Some machine instructions are priviliged and result in an error when executed in user mode

Lecture 2:Introduction to the KERNEL

UNIX system supports the illusion that:• File system has “places” &• Processes have “life”

Two central concepts in the UNIX system model!!

KERNEL• The kernel of UNIX is the hub of the operating system: It

allocates time and memory to programs and handles the filestore and communications in response to system calls.

• As an illustration of the way the shell and the kernel work together, suppose a user types rm myfile (which has the effect of removing the file myfile). The shell searches the filestore for the file containing the program rm, and then requests the kernel, through system calls, to execute the program rm on myfile. When the process rm myfile has finished running, the shell then returns the UNIX prompt to the user, indicating that it is waiting for further commands.

Block Diagram of Kernel

Hardware

Hardware controlKernel Level Hardware Level

libraries

System call interface

File subsystem

Buffer cache

Character | block

Device drivers

process

control

subsystem

Inter-process communication

scheduler

Memory management

User programs

User Level

Kernel Level

File subsystem

• The internal representation of a file is given by an “inode”• File table is a global kernel structure but the user file descriptor

table is allocated per process (when a process opens or creates a file, the kernel allocates an entry from each table, corresponding to the file’s inode. These three entries maintain the state of the file and the user’s access to it

• A file system consists of a sequence of logical blocks each containing 512, 1024, 2048 or any convenient multiple of 512 bytes, depending on the system implementation

“block”: logical stock contains 1K bytes of data

Overview of the File Subsystem

User File Descriptor

Table

File Table Inode Table

File System Layout

• Boot block: occupies the beginning of a file system, typically the first sector and may contain the bootstrap code that is read into the machine to boot, or initialise the operating system

• Super block: describes the state of a file system. How large it is, how many files it can store, where to find free space on the file system and other information

• The inode list is a list of inodes. One inode is the root inode of the file system: it is the inode by which the directory structure of the file system is accessible after execution of the system call

• The data blocks start at the end of the inode list and contain file data and administrative data. All allocated data blocks can belong to one & only one file in the file system.

Boot block Super block Inode list Data blocks

Process Process is the execution of a program and consists of

a pattern of bytes that the CPU interprets as machine instructions, data and stacks.

• A process executes by following a strict sequence of instructions that is self-contained and does not jump to that of another process;

• It reads and writes its data and stack sections, but it cannot read or write the dats and stack of other processes;

• Processes communicate with other processes and with the rest of the world via system calls.

Process (cont.)

• In practical terms, a process on a UNIX system is the entity that is created by the fork system call. Every process except process 0 is created when another process executes the fork system call. The process that invoked the fork system call is the parent process and th enewly created is the child process.

• Every process can be identified by the kernel by its process number, called the process ID (PID).

• Process 0 is a special process that is created “by hand” when the system boots.

Process (cont.)

A user compiles the source code of a program to create an executable file, which consists of several parts:

• A set of “headers” that describe the attributes of a file• The program text• A machine language representation of data that has

initial values when the program starts execution, and an indication of how much space the kernel should allocate for uninitialized data

• Other sections, such as symbol table information

Process (cont.)

The kernel loads and executable file into memory during an exec system call and the loaded process consists of at least three partscalled regions

• text, data correspond to sections of executable file and • the stack automatically created and its size is dynamically

adjusted by the kernel at run time

Stack:

a. Logical stack of frames that are pushed when calling a function and popped when returning

b. Stack pointer: indicates the current stack depth

c. Stack frame: parameters to a function, local variables, data necessary to recover the previous stack frame

User and Kernel Stack for copy program

Local not

Vars shown

Direction of stack growth

Addr of Frame 2

Ret addr after write call

Parmss to write

(new buffer count)

Frame 3

Call write() Frame 3

Local count

Vars

Local

Vars

Addr of Frame 1 Addr of Frame 1

Ret addr after copy call Ret addr after func2 call

Parms to copy (old new) Frame 2 Frame 2 Parms to kernel func2

Local fdold

Vars fdnew

call copy() call func2() Local

Vars

Addr of Frame 0 Addr of Frame 0

Ret addr after main call Ret addr after func1 call

Parms to main (argc argv) Frame 1 Frame 1 Parms to kernel func1

User Stack Kernel Stack

call main()

Frame 0

Start

call func1()

Frame 0

System call interface

Process (cont.)

• Every process has an entry in the kernel process table, and each process is allocated a u area that contains private data manipulated only by the kernel

• The process table contains a per process region table, whose entries point to entries in a region table

• A region is a contiguous area of a process’s address space, such as text, data and stack

• Region table entries describe the attributes of the region, such as whether it contains text or data, whether it is shared or private and where the data of the region is located in the memory

• The process table points to a per process region table with pointers to the region table entries for the text, data and stack regions of the process

Data structures for processes

The u-area contains information describing the process that needs to be accessible only when the process is executing:

• A pointer to the process table slot of the currently executing process

• Parameters of the current system call, return values or error codes

• File descriptors for all open files• Internal I/O parameters• Current directory and current root• Process and file size limits

u area

Main memoryProcess table

Per process region table

region table

Process states (cont.)

The lifetime of a process can be divided into a set of states, each with certain characteristics that describe the process:

• The process is currently executing in user mode• The process is currently executing in kernel mode• The process is not executing but it is ready to run as

soon as the schedule chooses it. Many processes may be in this state and the scheduling algorithm determines which one will execute next

• The process is sleeping. A process puts itself to sleep when it can no longer continue executing, such as when it is waiting for I/O to complete

State transitions

• Process move continuously between states according to well-defined rules.

• A state transition diagram is a directed graph whose nodes represent the states a process can enter and whose edges represent the events that cause a process to move from one state to another.

• State transitions are legal between two states if there exists an edge from the first state to the second.

• Several process can execute simultaneuously in a time-shared manner and they may all run simultaneuously in kernel mode.

Process states & Transitions

1

2

34

User running

Sys call or interrupt

return Interrupt, interrupt return

Schedule process

Ready to runWake up

sleep

Asleep

Context switch permissible