35
Advanced Operating Advanced Operating Systems Systems Prof. Muhammad Saeed File File Systems-I Systems-I

Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Embed Size (px)

Citation preview

Page 1: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Advanced Operating SystemsSystems

Prof. Muhammad Saeed

File Systems-IFile Systems-I

Page 2: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 2

File SystemsFile Systems

FilesDirectories & namingFile system implementationExample file systems

Page 3: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 3

Long-Term Information StorageLong-Term Information StorageMust store large amounts of data

Gigabytes -> terabytes -> petabytes

Stored information must survive the termination of the process using it (persistence)

Lifetime can be seconds to yearsMust have some way of finding it!

Multiple processes must be able to access the information concurrently

Page 4: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 4

Naming filesNaming files

Important to be able to find files after they’re createdEvery file has at least one nameName can be

Human-accessible: “solve.c”, “my drawing”, “Air Routes!”, “Speech Final Draft”Machine-usable: 4502, 33481

Case may or may not matterDepends on the file system

Name may include information about the file’s contentsCertainly does for the user (the name should make it easy to figure out what’s in it!)Computer may use part of the name to determine the file type

Page 5: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 5

Typical file extensionsTypical file extensions

Page 6: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 6

File structuresFile structures

Sequence of bytes Sequence of records

1 byte1 record

12A 101 111

sab wm cm avg ejw sab elm br

S02 F01 W02

Tree

Page 7: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 7

File typesFile types

Executablefile

Archive

Page 8: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 8

Accessing a fileAccessing a file

Sequential accessRead all bytes/records from the beginningCannot jump around

May rewind or back up, however

Convenient when medium was magnetic tapeOften useful when whole file is needed

Random accessBytes (or records) read in any orderEssential for database systemsRead can be …

Move file marker (seek), then read or …Read and then move file marker

Page 9: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 9

File attributesFile attributes

Page 10: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 10

File operationsFile operationsCreate: make a new fileDelete: remove an existing fileOpen: prepare a file to be accessedClose: indicate that a file is no longer being accessedRead: get data from a fileWrite: put data to a file

Append: like write, but only at the end of the fileSeek: move the “current” pointer elsewhere in the fileGet attributes: retrieve attribute informationSet attributes: modify attribute informationRename: change a file’s name

Page 11: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 13

Memory-mapped filesMemory-mapped files

Programtext

Data

Before mapping

Programtext

Data

After mapping

abc

xyz

Segmented process before mapping files into its address spaceProcess after mapping

Existing file abc into one segment Creating new segment for xyz

Page 12: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 14

More on memory-mapped filesMore on memory-mapped files

Memory-mapped files are a convenient abstractionExample: string search in a large file can be done just as with memory!Let the OS do the buffering (reads & writes) in the virtual memory system

Some issues come up…How long is the file?

Easy if read-onlyDifficult if writes allowed: what if a write is past the end of file?

What happens if the file is shared: when do changes appear to other processes?When are writes flushed out to disk?

Clearly, easier to memory map read-only files…

Page 13: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 15

Virtual File SystemVirtual File System

Page 14: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 16

DirectoriesDirectoriesNaming is nice, but limitedHumans like to group things together for convenienceFile systems allow this to be done with directories (sometimes called folders)Grouping makes it easier to

Find files in the first place: remember the enclosing directories for the fileLocate related files (or just determine which files are related)

Page 15: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 17

Single-level directory systemsSingle-level directory systems

One directory in the file systemExample directory

Contains 4 files (abc, def, ghi, jkl)owned by 3 different people: A, B, and C (owners shown in red)

Problem: what if user B wants to create a file called abc?

Rootdirectory

Aabc

Adef

Bghi

Cjkl

Page 16: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 18

Two-level directory systemTwo-level directory system

Solves naming problem: each user has her own directoryMultiple users can use the same file nameBy default, users access files in their own directoriesExtension: allow users to access files in others’ directories

Rootdirectory

Afoo

Abar

Bfoo

Bbaz

A B C

Cbar

Cfoo

Cblah

Page 17: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 19

Hierarchical directory systemHierarchical directory systemRoot

directory

Afoo

AMom

Bfoo

Bfoo.tex

A B C

Cbar

Cfoo

Cblah

APapers

APhotos

AFamily

Asunset

Asunset

Aos.tex

Akids

BPapers

Bfoo.ps

Page 18: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 20

Unix directory treeUnix directory tree

Page 19: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 21

Operations on directoriesOperations on directories

Create: make a new directoryDelete: remove a directory (usually must be empty)Opendir: open a directory to allow searching itClosedir: close a directory (done searching)

Readdir: read a directory entryRename: change the name of a directory

Similar to renaming a file

Link: create a new entry in a directory to link to an existing fileUnlink: remove an entry in a directory

Remove the file if this is the last link to this file

Page 20: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 22

File system implementation issuesFile system implementation issues

How are disks divided up into file systems?How does the file system allocate blocks to files?How does the file system manage free space?How are directories handled?How can the file system improve…

Performance?Reliability?

Page 21: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 23

Carving up the diskCarving up the disk

Masterboot record

Partition table

Partition 1 Partition 2 Partition 3 Partition 4

Entire disk

Bootblock

Superblock

Free spacemanagement

Indexnodes Files & directories

Page 22: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 24

Contiguous allocation for file blocksContiguous allocation for file blocks

Contiguous allocation requires all blocks of a file to be consecutive on diskProblem: deleting files leaves “holes”

Similar to memory allocation issuesCompacting the disk can be a very slow procedure…

A B C D E F

A Free C Free E F

Page 23: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 25

Contiguous allocationContiguous allocation

Data in each file is stored in consecutive blocks on diskSimple & efficient indexing

Starting location (block #) on disk (start)Length of the file in blocks (length)

Random access well-supportedDifficult to grow files

Must pre-allocate all needed spaceWasteful of storage if file isn’t using all of the spaceLogical to physical mapping

is easyblocknum = (pos / 1024) + start;offset_in_block = pos % 1024;

Start=5Length=2902

0 1 2 3

4 5 6 7

8 9 10 11

Page 24: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 26

Linked allocationLinked allocation

File is a linked list of disk blocks

Blocks may be scattered around the disk driveBlock contains both pointer to next block and dataFiles may be as long as needed

New blocks are allocated as needed

Linked into list of blocks in fileRemoved from list (bitmap) of free blocks

0 1 2 3

4 5 6 7

8 9 10 11

Start=9End=4Length=2902

Start=3End=6Length=1500

0

x

4 6

x

Page 25: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 27

Finding blocks with linked allocationFinding blocks with linked allocation

Directory structure is simpleStarting address looked up from directoryDirectory only keeps track of first block (not others)

No wasted space - all blocks can be usedRandom access is difficult: must always start at first block!Logical to physical mapping is done byblock = start;offset_in_block = pos % 1024;for (j = 0; j < pos / 1024; j++) { block = block->next;}

Assumes that next pointer is stored at end of blockMay require a long time for seek to random location in file

Page 26: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 28

Linked allocation using a Linked allocation using a RAM-based tableRAM-based table

Links on disk are slowKeep linked list in memoryAdvantage: fasterDisadvantages

Have to copy it to disk at some pointHave to keep in-memory and on-disk copy consistent

Page 27: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 29

Using a block index Using a block index for allocationfor allocation

Store file block addresses in an array

Array itself is stored in a disk blockDirectory has a pointer to this disk blockNon-existent blocks indicated by -1

Random access easyLimit on file size?

0 1 2 3

4 5 6 7

8 9 10 11

grades 4 4802Name index size

69708

Page 28: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 30

Finding blocks with indexed allocationFinding blocks with indexed allocation

Need location of index table: look up in directoryRandom & sequential access both well-supported: look up block number in index tableSpace utilization is good

No wasted disk blocks (allocate individually)Files can grow and shrink easilyOverhead of a single disk block per file

Logical to physical mapping is done byblock = index[block % 1024];offset_in_block = pos % 1024;

Limited file size: 256 pointers per index block, 1 KB per file block -> 256 KB per file limit

Page 29: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 31

Larger files with indexed allocationLarger files with indexed allocation

How can indexed allocation allow files larger than a single index block?Linked index blocks: similar to linked file blocks, but using index blocks insteadLogical to physical mapping is done byindex = start;blocknum = pos / 1024;for (j = 0; j < blocknum /255); j++) { index = index->next;}block = index[blocknum % 255];offset_in_block = pos % 1024;

File size is now unlimitedRandom access slow, but only for very large files

Page 30: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 32

Two-level indexed allocationTwo-level indexed allocation

Allow larger files by creating an index of index blocksFile size still limited, but much largerLimit for 1 KB blocks = 1 KB * 256 * 256 = 226 bytes = 64 MB

Logical to physical mapping is done byblocknum = pos / 1024;index = start[blocknum / 256)];block = index[blocknum % 256]offset_in_block = pos % 1024;

Start is the only pointer kept in the directoryOverhead is now at least two blocks per file

This can be extended to more than two levels if larger files are needed...

Page 31: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 33

Block allocation with extentsBlock allocation with extentsReduce space consumed by index pointers

Often, consecutive blocks in file are sequential on diskStore <block,count> instead of just <block> in indexAt each level, keep total count for the index for efficiency

Lookup procedure is:Find correct index block by checking the starting file offset for each index blockFind correct <block,count> entry by running through index block, keeping track of how far into file the entry isFind correct block in <block,count> pair

More efficient if file blocks tend to be consecutive on disk

Allocating blocks like this allows faster reads & writesLookup is somewhat more complex

Page 32: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 34

The ext4 filesystem can support volumes with sizes up to 1 exbibyte (EiB) and files with sizes up to 16 tebibytes (TiB). Stable system in 2008.

Extents replace the traditional block mapping scheme used by ext2/3 filesystems. An extent is a range of contiguous physical blocks, improving large file performance and reducing fragmentation. A single extent in ext4 can map up to 128 MiB of contiguous space with a 4 KiB block size.1 There can be 4 extents stored in the inode. When there are more than 4 extents to a file, the rest of the extents are indexed in an Htree.

ExtentsExtents

Page 33: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 35

The Relationships Among Segments, The Relationships Among Segments, Extents, and Data Blocks as implemented by Extents, and Data Blocks as implemented by OracleOracle

Page 34: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 36

FS Name Year Introduced

Original OS Max File Size Max FS Size Journaling

FAT16 1983 MSDOS V2 4GB 16MB to 8GB N

FAT32 1997 Windows 95 4GB 8GB to 2TB NHPFS 1988 OS/2 4GB 2TB NNTFS 1993 Windows NT 16EB 16EB YHFS+ 1998 Mac OS 8EB ? NUFS2 2002 FreeBSD 512GB to

32PB1YB N

ext2 1993 Linux 16GB to 2TB4 2TB to 32TB N

ext3 1999 Linux 16GB to 2TB4 2TB to 32TB Y

ReiserFS3 2001 Linux 8TB8 16TB YReiserFS4 2005 Linux ? ? YXFS 1994 IRIX 9EB 9EB YJFS ? AIX 8EB 512TB to 4PB Y

VxFS 1991 SVR4.0 16EB ? YZFS 2004 Solaris 10 1YB 16EB N

Page 35: Advanced Operating Systems Prof. Muhammad Saeed File Systems-I

Advanced Operating Systems 37

ENDEND

Courtesy of University of PITTSBURGH