File Systems• We need a mechanism that provides long-

term information storage with following characteristics:

1. Possible to store large amount of INFO2. INFO survives after termination of any

process3. Multiple processes can access INFO

concurrently• The file system is the component of O.S. that

manipulate the INFO as files and directories• The file systems is the appearance of INFO

from the user’s standpoint that involved two main structures :Files and directories

Files

• INFO stored in the files must be persistent, that is, not be affected by process creation and termination

• A file is a logical storage unit defined by the O.S. providing the user a mechanism to store INFO on a physical storage devices such as disk , tape , CD and etc.

user O.S. Physical

Logical View view------------

File Naming

• Some O.S. recognize difference between upper and lower case letters ( e.g., Unix) and some of them don’t (e.g., MS-DOS)

• The file extension usually indicates what type of file it is (see the next slide). In some systems (e.g., UNIX), file extension are just conventions and are not enforced by O.S. Some other systems (e.g., Windows) are aware of extension and use programs that are assigned to the extensions (e.g., file.doc starts Word)

File Structure

• The structure of a file is determined by O.S.• Some O.S.,’s (e.g., CPM and old mainframes)

impose the view that a file is a sequence of fixed length records ( e.g., b in the next slide)

• Other O.S.’s may impose a B-tree (or index) like structure on a file in order to support rapid search ( e.g., c in the next slide)

• The problem with imposing more structure by O.S. is it is difficult to do something out of the ordering that is not foreseen by O.S. designer

File Structure

• O.S. systems such as UNIX and Windows impose no structure to ensure maximum flexibility. They consider a file as a steam of bytes , and user processes define any structure that they want

• I/O is usually performed in units of ONE physical Block and all blocks have the same size that is related to the page size in paging scheme.

File Types

Some of the file types are• Regular files: User files (ASCII files or binary

files) • Directory files: System files used to maintain

directory structure• I/O files: Special system files dedicated to I/O• Executable files: O.S. usually expects special

structure for these files. For example in Unix they must start with Magic Number. Next slide shows difference between executable (a) and archive (i.e. compiled but not linked) file in Unix

File Access

Generally two types of access are provided for the files :

• Sequential access: starts from the beginning and read sequentially (usually is using with tapes)

• Random access: can access any byte in the file directly.

O.S. provides these operations to the user

File Attributes

Deals with:• Location: where the file is physically located• Size: how big is the file• Type : what kind of file it is• Protection: who can access the file• Time & Date: when was the last access or

modification• User: who created the file and other information. Some of the attributes are

shown in the next slide

File OperationsMost common system calls relating to files• Create: announce that file is coming and

set attributes and allocate space• Delete: Free disk space, adjust directory

structure• Open : Fetch the attributes and location of

the file• Close: Release internal table space and

writing the file’s last block

File Operations

• Read: Data read from the file and put into memory for user access

• Write: Data are written to the file usually at the current position

• Append: Adds data to the end of file• Seek: Random access data from the file, repositioning the

file pointer for reading• Rename: Change the name of the file• Get & Set attribute: Get attributes of file or set attributes of

a file (e.g., get and set read only attribute ) See the program for copying a file in UNIX shown in the next

slides. It can be called by the following command line: copyfile abc xyz

Directories

• Directories are the mechanism provided by O.S. to keep track of files. A directory records info a bout the files in the particular partition.

• Directory typically contains one entry per file. It may contain Name, Attributes and Location or

• It may contain Name and pointer to Attribute information

Directory Structure

• Single level directory system• No owner, problem is the files with the same names

created by two different owners• Note that in the following Figures the files are shown by

the owner names. For example the files named A created by the same owner.

Directory Structure• Two-level directory system• Search in directories is based on user name.

Problem is the user with the large number of files

Directory Structure

• Hierarchical directory system

Path Names

• Absolute path name: /usr/ast/mailbox. Always starts with / (i.e.,separator)

• Relative Path Name: mailbox

• Current directory or working directory determines the relative path name

• In Unix . is current directory and

.. refers to parent

• For example: cp ../lib/abdy.doc .

Directory Operations

• Create : creates . , ..• Delete : only empty directory can be deleted• Rename• Link & unlink: link is a common technique used

for sharing files or directories between users. (see next slide). Instead of link, duplication of the files can be used for shared files but the problem of duplication is consistency is difficult to maintain. Link within a directory can be hard link (implemented by i-node that explained later) or symbolic linking (creating a file that contains the path of the linked file).

Directories• Creating a shared file by link changes the

directory structure from a tree to a graph

File System Layout• Most disks divided up into one or more partitions,

with independent file systems on each partition.• Sector 0 of disk is called MBR ( Master Boot

Record) and contains partition table that contains start and ending address for each partition

• The layout of a disk partition depends on its file system. For example after its first block ( i.e., boot block) it may contain super block that contains administrative information such as magic numbers to identify file types. (see next slide)

Implementing the FilesVarious methods are used in different O.S. for

implementing the files:• Contiguous Allocation: Each file is stored on

consecutive disk blocks. For example for a disk with 4K block size a 20K file is stored on 5 consecutive blocks. (see next slide)

Advantages: • simple to implement because we need to know

only disk address of the first block and number of blocks

• The read performance is excellent because we need only one disk operation to read the entire file.

Contiguous Allocation

Contiguous AllocationThe disadvantages of Contiguous allocation are:• Disk fragmentation: happens when the files are

removed. Compaction is difficult because all the blocks following the holes should be copied. It is worse when the disk filled up.

• Needs to know the final size of new file to be able to choose the correct hole to place it. That is also difficult

Consecutive allocation is good for write once medias such as CD-ROMS and DVDs

Linked List Allocation

• A linked list of disk blocks (first word is pointer) is kept in this method

• Every disk blocks can be used (except for internal fragmentation)

• The sequential read for the blocks of the file is easy but random access to each block is hard because we have to read all the blocks of a file before that block

• Because of pointer the amount of data stored in each block is not a power of two

Linked List Allocation

Linked List Allocation using a Table in

Memory • Both of disadvantages of the linked list allocation

can be eliminated by keeping the table of pointer to the blocks (FAT) in the memory. MSDOS uses that.

• Random access to blocks is easy because there is no disk reference involved. We need only the starting block number.

• The problem is for 20 GB disk, and a 1 KB block size table needs 20 million entries if each be 4 bytes, table will take approximately 80 MB .

File Allocation Table

I-nodes

• To solve the problem of the large file table we can use i-node

• In this method for each file there is a table contains attributes and disk address of the blocks of that file. So if i-node occupies n bytes for k files open we have kn bytes of memory. Thus i-node depends on open files not disk size

• Problem is if each i-node has room for a fixed number of disk addresses what happens when a file grows beyond this limit?

• One solution is keeping multiple indexes in i-node.

I-node in Unix

i-node in UNIX has • Initial 10 disk addresses. • Single indirect blocks keeps address of file

more blocks for larger files.• Double indirect block that holds address of

the blocks each contains a list of single indirect block

• Triple indirect block has the address of block each is double indirect block

I-node in Unix

Implementing Directories

• Basically, a directory is a file that contains an entry for each file or subdirectory in that directory

• When a file is opened, O.S. uses the path name to locate directory entry

• Each directory entry contain the file information• Each file information can be stored directly in

directory entry (a in the next slide) • Or file information can be stored in i-node and

each directory entry refers to i-node (b in the next slide)

Implementing Directories

Directories in MS-DOS

• Same as CP/M directory entries they are 32 bits each

• The extension is for a large file size that requires more than one directory entry. The order in which directory entries should be followed

• First block number is the physical block number address of the file

Directories in MS-DOS

Directories in UNIX

• Each directory entry contains file name and i-node number

Directories in UNIX

• Directory lookup in Unix and all hierarchical system is same

• First file system locates the root directory.• Then it looks up the first component of the path

and its i-node• From the i-node system looks up the block

address of next component and it works in the same way until the file can be found. For example next slide shows the steps in looking up /usr/ast/mbox

Disk Space ManagementPhysical Disk Structure

• Main secondary storage is disk. Tape mainly is used for backup

• The physical disk consists of cylinders. Each cylinder is divided into tracks. A track is divided further into sectors. One or more sectors form a logical block. Data transfers between the main memory and disk are in the units of logical blocks. The size of a logical block is usually 512 bytes or larger, although the disk can be formatted to have different logical block sizes

Disk Read Speed

• The total time for accessing a file consists of the time to move the head to the right track (seek time), the time to find a correct sector (rotational delay), and the time to transfer data (transfer time). Disk seek time contributes more to the total delay of accessing the files, especially when files are not stored in contiguous blocks.

Disk Read• Example : The seek time is 10 msec per block in

average, and rotation latency is 8 msec per block in average and transfer time is 0.25 msec for 1KB block for a disk system. The average reading time for each block in this disk system is: 10 + 8 + 0.25 = 18.25 ms

• Usually as shown in this example seek time and rotation time contribute more to disk read latency.

• It means if we reduce seek time or rotation latency we can increase disk read time significantly. Therefore most of the optimizations for increasing disk performance are based on reducing disk seek time.

• For example in Unix FFS uses cylinder grouping technique to reduce disk seek time

Cylinder Grouping Technique

• Fast File System (FFS, a Unix file system) uses the cylinder grouping technique to provide both block-level and file-level clustering. In the cylinder grouping technique, users or applications have to place the related files into a directory. The files of the directory are allocated in one or more consecutive cylinders to reduce disk seek time (see next slide). In the cylinder grouping technique, files belonging to a directory are stored on consecutive blocks on disk(s). With the same approach, FFS also tries to store a single file in consecutive disk blocks.

Keeping Track of Free Blocks There are two methods for keeping track of

the free disk blocks. Linked list and bitmap

• Often free blocks on disk can be used to hold the number of free blocks. For example (a) in the next slide shows three free blocks (16,17 and 18) that maintain the block numbers of the free blocks with linked list method.

(a)(b)

Free disk blocks: 16, 17 , 18

Keeping Track of Free Blocks

• In the bit map method one bit required for each block, where 1 shows block is used and 0 shows the block is free. Bit map method requires less space compare to linked list, except for the situation in which disk is full and there is only free few blocks on disk.

File System Reliability

• Bad block management: Most hard disk have bad blocks that can be resolved by hardware solution or software solution

File System Reliability

Backups:

Full backups

• Problem: taking long time and space.

• Solution: instead of the entire file system only part of that can be backed up. There is no reason to backup /bin or /dev files in UNIX

File System Reliability

Incremental dumps: to make a complete dump (backup) periodically and make daily backup of only those files that have been modified since the last dump

• Advantage: minimize the backup time

• Disadvantage: It makes recovery more complicated

File System Consistency

• If the system crashes before writing all the modified blocks, file system becomes inconsistent.

• Solution: Checking the file system consistency. For example fsck in UNIX or scandisk in Windows

File System Consistency• Two type of consistency checks can be

made: block and files consistency checkBlock consistency check:• Two tables are builds each contains a

counter for all blocks • Program reads all i-nodes to find used

blocks and updates first table• Program examines free list/bit map to find

not used blocks and updates second table

Block Consistency Check

Consistent Missing block

Duplicate block in free list Duplicate data block

Block number

File Inconsistency CheckCan be done by• Using a table of counters per file.• Verifying directory system by traversing the

directory tree. It can be done by incrementing the counter for each file based on the number of time that file has been used in the directories

• Comparing the number of file usage with the link count (i.e., a number reported by i-node of that file) shows the consistency/inconsistency

File System Performance

• Access to disk is much slower than access to memory. In memory reading a word takes 10 nsec

• Solution: Using block cache buffer in the memory

• For each read request, cache is checked for availability of the requested block

Caching

• Cache references are less than paging so using LRU for cache is feasible

• Disadvantage of using LRU is a crash will leave file system inconsistence

Buffer Cache Data Structure

CachingSolution:

• The needed blocks such as i-node and directory can put at the front (to be evicted faster) instead of rear. It means they can be written on disk more frequently. This reduces the chance of inconsistency in file system.

• Writing modified data blocks immediately. Sync in UNIX and write-through cache in MS-DOS can do that.

Block Read Ahead• It is the second technique for improving the file

performance• Reading ahead the blocks on each file read.

Only good for sequential file reads• Solution: Keeping access pattern of file by using

a bit for that file. By setting that bit in each sequential access and resetting in each random access (i.e., seek is done) system can guess if the file is in sequential or random access mode.

Reducing Arm Motion• Placing i-nodes in the middle of the disk

instead of start of the disk (see the next slide)

• Cylinder grouping technique (i-nodes and related files are in the same cylinder group)

Log-Structured File System

• Log-structured (or journaling) file system designed in Berkeley for UNIX to reduce disk seek times for the write operations

• In UNIX most of the write operations are small writes

Log-Structured File System• LFS considers the entire disk as a log and

by buffering the writes in the memory, writes them in a single segment at the end of log periodically.

• Each segment may contain i-nodes, directory entry blocks and data blocks

• The problem is i-nodes are scattered all over the log instead of being in the fixed disk position

Log-Structured File System

• Opening a file consists of using map to locate the i-node for that file

• LFS has a book keeping program named cleaner that moves around the log and remove old segments

The Sun Network File System (NFS)• The implementation is part of the Solaris

and SunOS operating systems running on Sun workstations using an unreliable datagram protocol (UDP/IP protocol and Ethernet.

• NFS is designed to operate in a heterogeneous environment

• In NFS clients access the server directories by mounting them

Remote Mounting in NFS

Remote Mounting in NFS

Remote Mounting in NFSMount operation includes name of remote

directory to be mounted and name of server machine that is storing it. • Mount request is mapped to corresponding

RPC and forwarded to mount server running on server machine.

• Export list – specifies local file systems that server exports for mounting, along with names of machines that are permitted to mount them.

Remote Mounting in NFS

• Following a mount request that conforms to its export list, the server returns a file handle—a key for further accesses.

• File handle – a file-system identifier, and an i-node number is used to identify the mounted directory within the exported file system.