Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP ’03

CSC 456 Operating Systems Seminar Presentation (11/13/2012)

Leon Weingard, Liang Xin

The Google File System

Outline1. Background2. Distributed File Systems Overview3. GFS Architecture4. Fault Tolerance

Background - Introduction

• Google – search engine.• Applications process lots of data.• Need good file system.• Solution: Google File System (GFS).

IntroductionThe largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients.

Background - Motivation

Goal:

Large, distributed, highly fault tolerant file system

1. Fault-tolerance and auto-recovery need to be built into the system.2. Standard I/O assumptions (e.g. block size) have to be re-examined.3. Record appends are the prevalent form of writing.4. Google applications and GFS should be co-designed.

Distributed File Systems

Performance Measurement of a DFS depends on :

• The amount of time needed to satisfy service requests.• The multiplicity and dispersion of its servers and storage devices

should be made invisible.

In computing, a distributed file system is any file system that allows access to files from multiple hosts sharing via a computer network.

Network File System (NFS)

Network File System (NFS) is a distributed file system protocol originally developed by

Sun Microsystems in 1984.

• Architecture:– NFS as collection of protocols the provide clients with a distributed file system.– Remote Access Model (as opposed to Upload/Download Model)– Every machine can be both a client and a server.– Servers export directories for access by remote clients – Clients access exported directories by mounting them remotely.

• Protocols:– mounting – file and directory access

Workgroup network file service Any Unix machine can be a server (easily) Machines can be both client & server

My files on my disk, your files on your disk Everybody in group can access all files

Serious trust, scaling problems

The Andrew File System (AFS)The Andrew File System was introduced by researchers at Carnegie-Mellon University in the 1980’s.The basic tenets of all versions of AFS is whole-file caching on the local disk of the client machine that is accessing a file.

Goal: Scalability

Features:

Uniform name space Location independent file sharing Client side caching with cache consistency Secure authentication by Kerberos Scalability “No write conflict” model only partial success

All the files in AFS are distributed among the servers. The set of files in one server is referred to as a volume. In case a request can not be satisfied from this set of files, the vice server informs the client where it can find the required file.

Differences between AFS and NFS

NFSDistributed with OSclient side cache is optionalclear-text passwords on netDoes not scale wellUses standard UNIX

permissionsNot secureMore reliable than AFS?

AFSAdd-in productclient side caching is standardauthenticated challenge on netscales wellUses Access Control Lists

(ACL’s)More secure than NFSLess reliable than NFS?

GFS ArchitectureIn the GFS:

A master process maintains the metadata.

A lower layer (i.e. a set of chunkservers) stores the data in units called “chunks”.

GFS ArchitectureWhat is a master?

A single process running on a separate machine.

• Stores all metadata• File namespace• File to chunk mappings• Chunk location information• Access control information• Chunk version numbers…

GFS Architecture

What is a chunk?

Analogous to block, except larger.Size: 64 MB!Stored on chunkserver as fileChunk handle (~ chunk file name) used to reference chunk.Chunk replicated across multiple chunkserversNote: There are hundreds of chunkservers in aGFS cluster distributed over multiple racks.

Read Algorithm

Read Algorithm

Primary and Leases Master grants a chunk lease to one replica. Replica is called the primary. Primary picks an order for mutations to the chunk. Leases expire in 60 seconds. Primary can request an extension if chunk is being mutated. Master can revoke a lease before it expires. Master can assign a new lease if it loses contact with the primary.

Write Algorithm

Write Algorithm

Write Algorithm

Write Algorithm

Fault Tolerance: Chunks

Fast Recovery: master and chunkservers are designed to restartand restore state in a few seconds.No persistent log of chunk location in the master.Syncing chunkservers and master is unnecessary under this model.Alternate view: Chunkservers hold final say on what chunks it has on its disk.Chunk Replication: across multiple machines, across multipleracks.

Fault Tolerance: Master

Data structures are kept in memory, must be able to recover from system failure.Operation Log: Log of all changes made to metadata.Log records batched before flushed to disk.Checkpoints of state when log grows to big.Log and latest checkpoint used to recover state.Log and checkpoints replicated on multiple machines.Master state is replicated on multiple machines.“Shadow” masters for reading data if “real” master is down.Data integrity:Each chunk has an associated checksum.

summaryDistributed File Systems• NFS• AFSDifferences between AFS and NFS GFS ArchitectureSingle master with metadata and many chunk servers.GFS designed to be fault tolerant. Master must be able to recover. chunk locations not persistentMost operations are reads and appends so record append added so append operations can be done efficiently

(These slides modified from Alex Moshchuk, University of Washington – used during Google lecture series.)

References 1. The Google File System, SOSP ’03 2. Presentation slide of GFS in SOSP ’03 3. www.delmarlearning.com/companions/content/.../Ch18.ppt 4. lyle.smu.edu/cse/8343/fall_2003/.../DFS_Presentation.ppt

Thank you!