A Flash-Memory Based File System

(To Appear in USENIX ’95 Winter) 1 (Page Number will be erased in the final version)

A Flash-Memory Based File System

Atsuo Kawaguchi, Shingo Nishioka, and Hiroshi MotodaAdvanced Research laboratory, Hitachi, Ltd.

AbstractA flash memory device driver that supports a

conventional UNIX file system transparently wasdesigned. To avoid the limitations due to flashmemory's restricted number of write cycles and itsinability to be overwritten, this driver writes data to theflash memory system sequentially as a Log-structuredFile System (LFS) does and uses a cleaner to collectvalid data blocks and reclaim invalid ones by erasingthe corresponding flash memory regions.Measurements showed that the overhead of the cleanerhas little effect on the performance of the prototypewhen utilization is low but that the effect becomescritical as the utilization gets higher, reducing therandom write throughput from 222 Kbytes/s at 30%utilization to 40 Kbytes/s at 90% utilization. Theperformance of the prototype in the AndrewBenchmark test is roughly equivalent to that of the4.4BSD Pageable Memory based File System (MFS).

1. IntroductionFlash memory, a nonvolatile memory IC

(Integrated Circuit) that can hold data without powerbeing supplied, is usually a ROM (Read OnlyMemory) but its content is electrically erasable andrewritable. The term “flash” is used to indicate that itis a whole chip or a block of contiguous data bytes (Wecall this block an erase sector). Many kinds of flashmemory products [1][2] are available, and theircharacteristics are summarized in Table 1†.

Because flash memory is five to ten times asexpensive per megabyte as hard disk drive (HDD)memory, it is not likely to become the main massstorage device in computers. Its light weight, lowenergy consumption, and shock resistance, however,

†There is another type of flash memory that has much smaller erasesectors. (See Section 4.)

make it very attractive for mass storage in portablecomputers. In fact, flash memory in the form of anIC-card commonly replaces the HDD or is used forauxiliary storage in portable personal computers.

Flash memory has two other disadvantageslimiting its use in computer systems. One is that itscontent cannot be overwritten: it must be erased beforenew data can be stored. Unfortunately, this eraseoperation usually takes about one second. The otherdisadvantage is that the number of erase operations fora memory cell is limited, and upper limits on the orderof 100 000 erasures are not unusual. An advantage offlash memory, however, is that its read speed is muchgreater than that of a HDD. The performance of flashmemory in read operations is, in fact, almostequivalent to that of conventional DRAM.

Our objective in the work described here was toexplore the possibilities of using flash memory in filesystems and to develop an experimental but practicalflash memory based file system for UNIX. We used alog approach to ensure that new data was alwayswritten in a known location—so that the eraseoperation could be performed in advance.

2. Design and ImplementationWe have designed and implemented a flash

memory device driver that emulates a HDD andsupports a standard UNIX file system transparently.We chose the device driver approach for its simplicity.

Read Cycle 80 - 150 nsWrite Cycle 10 µs/byteErase Cycle 1 s/block

Cycles limit 100 000 timesSector size 64 Kbytes

Power Consumption 30 - 50 mA in an active state20 - 100 µA in a standby state

Price 10 - 30 $/MByteTable 1. Flash memory characteristics.


Since flash memory can be accessed directly through aprocessor's memory bus, other approaches (such astightly coupling a flash memory system and a buffercache) might perform better by reducingmemory-to-memory copy operations. Such anapproach, however, would require a large number ofkernel modification because flash memory's erase andwrite properties differ greatly from those of the mainmemory.

2.1 OverviewOur driver must record which regions contain

invalid data and reclaim the invalid region by erasingthose regions. Furthermore, since the number of eraseoperations in each region is limited, the driver must atleast monitor the number in order to assure reliableoperation. In some cases, wear-leveling should beprovided.

Our driver maintains a sequential data structuresimilar to that of LFS [3][4]. It handles a write requestby appending the requested data to the tail of thestructure. To enable later retrieval of the data itmaintains a translation table for translating betweenphysical block number and flash memory address.Because the translation is made on the level of thephysical block number, our driver can be viewed, fromthe file-system aspect, as an ordinal block device.

When write operations take place the flashmemory system is fragmented into valid and invaliddata blocks. To reclaim invalid data blocks, we use acleaner that selects an erase sector, collects valid datablocks in the sector, copies their contents to the tail ofthe log, invalidates the copied blocks (andconsequently makes the entire sector invalid), andissues an erase command to reclaim the sector. Thefunctions of this cleaner are identical to those of LFS'scleaner. Our prototype does not implementwear-leveling, although it does maintain a log of eacherased sector.

2.2 Flash Memory CapabilityEarly generations of flash memory products

prohibit read or write operations while they areperforming a write or an erase operation. That is, whensuch a flash memory chip is performing an eraseoperation on an erase sector, data can neither be readfrom nor written to other erase sectors until the eraseoperation completes. A naive file systemimplementation ignoring this prohibition would not befeasible in a multitasking environment because itwould unpredictably block operations whenever theprogram wanted data from a flash memory chip that

was performing an erase operation triggered byanother program. This problem can be avoided bytemporarily caching in a buffer all valid data in theflash memory chip to be erased. This cachingoperation, however, could consume a significantamount of processor resources.

Recent flash memory products provide an erase-suspend capability that enables sectors that are notbeing erased to be read from during an erase operation.Some new products also support write operationsduring the erase suspension. Our driver assumes theunderlying flash memory system to be capable oferase- suspended read operations.

Flash memory generally takes more time for awrite operation than for a read operation. It provides awrite bandwidth of about 100 Kbytes/s per flashmemory chip, whereas a conventional SCSI HDDprovides a peak write bandwidth 10 to 100 timeshigher. Some recent flash memory productsincorporate page buffers for write operations. Thesebuffers enable a processor to send block data to a flashmemory chip faster. After sending the data, theprocessor issues a “Page Buffer Write” command tothe chip and the chip performs write operations whilethe processor does other jobs.

Our driver assumes that the underlying flashmemory system consists of some banks of memorythat support concurrent write operations on each chip.This assumption reduces the need for an on-chip pagebuffer because the concurrent write operations canprovide a higher transfer rate.

2.3 On Flash Memory Data StructureFigure 1 depicts our driver's data structure built on

an underlying flash memory system. The flashmemory system is logically handled as a collection ofbanks. A bank corresponds to a set of flash memorychips and each set can perform erase or write

bank 0

segment summary

bank 1 bank l• • • • • • •

• • • • • • • segment 0 segment 1 segment m

summary magic no.no. of blocks

block 0 informationblock 1 information

block n information

••••

physical block no.flag 0flag 1flag 2flag 3

block information entry

no. of erase op.

Figure 1. On-chip data structure.


operations independently. The banks are in turndivided into segments, each of which corresponds toan erase sector of the flash memory system. Eachsegment consists of a segment summary and an arrayof data blocks. The segment summary containssegment information and an array of block informationentries. Segment information includes the number ofblocks the segment contains and the number of timesthe segment has been erased.

2.4 Flag UpdateEach block information entry contains flags and

the physical block number to which this data blockcorresponds. The physical block number is provided tothe driver by the file-system module when issuing awrite data request. The flags are written sequentiallyso that the driver can record the change of the blockstatus without erasing the segment.

The driver uses four flags to minimize thepossibility of inconsistency and to make recoveryeasier. When a logical block is overwritten the driverinvalidates the old block, allocates a new block, andwrites new data to the newly allocated block. Thedriver updates the flags on the flash memory in thefollowing order:

Step 1.mark the newly allocated block asallocated,

Step 2.write the block number and then writenew data to the allocated block,

Step 3.mark the allocated block as pre-valid,Step 4.mark the invalidated block as invalid,

andStep 5.mark the allocated block as valid.

The above steps guarantee that the flag values ofthe newly allocated and invalidated blocks neverbecome the same under any circumstances. Therefore,even after a crash (e.g., a power failure) during any oneof the above steps, the driver can choose one of theblocks that hold the fully written data. This method isof course not sufficient to maintain complete filesystem consistency, but it helps suppress unnecessaryambiguity at the device level.

2.5 Bank ManagementTo manage block allocation and cleaning, the

driver maintains a bank list and a cleaning bank list.Figure 2 shows the relationship between these lists.The driver allocates a new data block from the activebank, so data write operations take place only on theactive bank. When the free blocks in the active bankare exhausted, the driver selects from the bank list thebank that has the most free segments (i.e., free blocks)and makes it the new active bank.

When a segment is selected to be cleaned, thebank containing that segment is moved to the cleaningbank list. The bank stays in the list until an erasureoperation on the segment finishes. Because the bank isno longer on the bank list, it never becomes an activebank, and thus avoids being written during the eraseoperation.

The driver maintains the flag update queue tohandle the flag update procedure, described in theprevious section, on blocks in the bank of whichsegment is being erased. The driver avoids issuing adata write on a bank being cleaned by separating thebank list and the cleaning bank list. However, when ablock is logically overwritten, an invalidated blockmight belong to that bank. In such a case, the driverpostpones the flag update procedure steps 4 and 5, byentering the pair of the newly allocated and theinvalidated blocks into the queue. All the pairs areprocessed when the erasure finishes. Note that even ifthe pairs are not processed due to a crash during theerasure, the driver can recover flag consistencybecause of the flag update order (Step 3 for each pairhas been completed before the crash occurs.).

The queue should be able to hold the number ofpairs that are expected to be entered during an erasure.For example, the current implementation can generate500 pairs for 1 erasure [i.e., 500 blocks (250 KBytes)per second], and thus has 600 entries in the queue.Should the queue be exhausted, the driver will stopwriting until the erasure is complete. We have not yetexperienced this condition.

2.6 Translation TableThe translation table data structure contains all

information needed to manage the translation of ablock number to an address and to manage the eraselog of each segment. During the system boot time, thedriver scans the flash memory system and constructsthis translation table and other structures from theon-chip segment summaries.

Figure 2. Bank list and cleaning bank list.

Bank listActivebank

••••

•••

Cleaningcandidates

Cleaning bank list

Flagupdatequeue


Figure 3 shows the relationship between thetranslation table and the block information entries.During the system boot time the driver scans all thesegment summaries one by one. If it finds a validblock, it records a triplet (bank no., segmentno., block no.) describing the block in a tableentry indexed by the physical block number.

After the boot, the driver refers only to thetranslation table to access data blocks on the flashmemory when a read operation is requested. Theaddress of each block can be computed from thetriplet. The driver translates a requested physical blocknumber to the address of a corresponding flashmemory data block and simply copies the contents ofthe data block to the upper layer.

When a write operation is requested, the driverchecks whether it has already allocated a flashmemory data block for a requested physical block. If ithas, the allocated block is invalidated. The driverallocates a new flash memory data block, updates thetranslation table, and copies the requested data to thenewly allocated block, while updating the flags.

2.7 CleanerThe segment cleaning operation takes place

during the allocation process when the number ofavailable flash memory blocks for writing becomeslow. This operation selects a segment to be cleaned,copies all valid data in the segment to anothersegment, and issues a block erase command for theselected segment. The cleaning process is the same asthat of LFS except that it explicitly invokes the eraseoperation on the segment.

Figure 3. Relationship between the block–address translation table and a block information entry.

Block©–addresstranslation

table

bank no. lsegment no. mblock no. n

i

physicalblock no. iflag 0,1,2,3

(Datablockfor i)

Summary

Datablockarray

0 ©•©•©•©•

©•©•©•©•

Segment m ofbank n

©•©•©•

©•©•©•

©•©•©•

©•©•

The cleaner is divided into three parts: policy,copying and erasing, and internal data maintenance.All jobs are executed in kernel-space, though copyingand erasing are conducted by a daemon running inuser-space.

As discussed in [4], implementing a cleaner as auser process makes the system flexible when changingor adding a cleaning policy or algorithm, or both. Bycommunicating with the kernel through the use ofsystem calls and the ifile, the BSD LFS cleaner doesalmost all jobs in user-space. Our driver, in contrast,does the cleaning jobs in kernel-space as Sprite LFSdoes. We use a daemon to make the copying processconcurrent with other processes. We took thisapproach for its ease of implementation.

While data is being written, cleaning policy codesare executed when a block is invalidated. If cleaningpolicy conditions are satisfied for a segment, the driveradds it to the cleaning list and wakes up the cleanerdaemon to start copying valid blocks. Uponawakening, the cleaner daemon invokes the copycommand repeatedly until all valid blocks are copiedto the clean segments. Then, it invokes the erasecommand and the driver starts erasing the segment byissuing an erase command of the flash memory. Thecopying is performed by codes within the driver. Thecleaning daemon controls the start of the copying. Itmakes the copying concurrent with other processes.

We added three ioctl commands for the cleanerdaemon (Table 2). The daemon first invokesFLIOCCWAIT and then waits (usually) until asegment to be cleaned is selected. As an applicationprogram writes or updates data in the file system, thedevice driver eventually encounters a segment thatneeds to be cleaned. The driver then wakes up thecleaner daemon and continues its execution.Eventually, the daemon starts running and invokesFLIOCCCBLK repeatedly until all the valid blocks arecopied to a new segment. On finishing the copyoperation, the daemon invokes FLIOCCIERS, whichcauses the driver to issue an erase command to theflash memory. The daemon invokes FLIOCCWAITagain and waits until another segment needs to becleaned.

Command DescriptionFLIOCCWAIT

FLIOCCCBLK

FLIOCCIERS

Wait until a segment is selected tobe cleaned.

Copy 16 valid blocks of the selectedsegment. Return 0 if no more validblocks exist in the segment .

Start erasing the segment. Return 0when the erasure is complete.

Table 2.ioctl commands for the cleaner.


2.8 Cleaning PolicyFor our driver, the cleaning policy concerns: • When the cleaner executes, and, • Which segments is to be cleaned.The flash memory hardware permits multiple

segments to be erased simultaneously as long as eachsegment belongs to the different bank. Thissimultaneous erasure provides a higher block-reclaimrate. For simplicity, however, the currentimplementation cleans one segment at a time. Thecleaner never tries to enhance logical block localityduring its copying activity. It simply collects andcopies live data in a segment being cleaned to a freesegment.

In order to select a segment to clean, the driver isequipped with two policies: “greedy” and“cost-benefit” [3] polices. The driver provides ioctlcommands to choose the policy. The greedy policyselects the segment containing the least amount ofvalid data, and the cost-benefit policy chooses themost valuable segment according to the formula:

where u is the utilization of the segment and age is thetime since the most recent modification (i.e., the lastblock invalidation). The terms 2u and 1-u respectivelyrepresent the cost for copying (u to read valid blocks inthe segment and u to write back them) and the freespace reclaimed. Note that LFS uses 1+u for thecopying cost because it reads the whole segment inorder to read valid blocks in the segment for cleaning.

The cleaning threshold defines when the cleanerstarts running. From the point of view of the load putupon the cleaner, the cleaning should be delayed aslong as possible. The delay should produce moreblocks to be invalidated and consequently reduce thenumber of valid blocks that must be copied during thecleaning activity. Delaying the cleaning activity toomuch, however, reduces the chances of the cleaningbeing done in parallel with other processes. Thisreduction may markedly slow the system.

Our driver uses a gradually falling threshold valuethat decreases as the number of free segmentsdecreases. The curve A in Figure 4 shows the thresholdof the current implementation. It shows that

• When enough (N) free segments are available, thecleaner is not executed,

• When the number of free segments becomessmaller than N and if there are some segmentswhose policy accounts are greater than Th,cleaning starts with the segment that has thegreatest policy accounts, and

benefitcost

age 1 u−( )×

2u,=

• As the number of free segments becomes smaller,the threshold becomes lower so that moresegments can be chosen with lower policyaccounts.

For the greedy policy of the current implementation, Nis 12 and Th is 455 invalid blocks. That is, when thenumber of free segments becomes 12, segments thatcontain more than 455 invalid blocks are cleaned. Forthe cost-benefit policy, N is 12 and Th is set to thevalue that is equivalent to being unmodified 30 dayswith one invalid block. For both the policies, segmentshaving no valid blocks are always cleaned before othersegments.

The threshold curve enables the driver to stop thecleaner as long as enough free segments are availableand also to start the cleaner at a slow pace. Forexample, suppose the driver employs a policy such as“When the number of free segments becomes Nb, startthe cleaner.” (This policy is represented by thethreshold curve B in Figure 4.) When the number offree segments became Nb the cleaner would startcleaning even if the most invalidated segment had onlyone invalid block. Furthermore, if the live data in thefile system counted more than Ns-Nb segments (whereNs is a total number of segments), the cleaner wouldrun every time a block was invalidated. This wouldresult in a file system that was impractically slow andhad an impractically short lifetime.

3. Performance Measurements and DiscussionUnlike a HDD-based file system, the prototype is

free from seek latency and it is thu expected to shownearly the same performance for both sequential andrandom read operations. In fact, for reading 4Kbyteblocks from 12.6 Mbytes of data, the sequential andrandom throughputs of the driver are respectively 644and 707 Kbytes/s. (For the same tasks, the throughthroughputs of MFS [7] are 647 and 702 Kbytes/s.)

0Ns N Nb 2 0

Th

B

A

Figure 4. Cleaning threshold.

Polic

y ac

coun

ts

Number of free segments


The write performance of the prototype, on the otherhand, is affected by the cleaning, as is the case withLFS.

The benchmark platform consists of ourhand-made computer running 4.4BSD UNIX with a40MHz R3081 processor and 64 Mbytes of mainmemory. The size of the buffer cache is 6 Mbytes.Table 3 summarizes the platform specification†.

3.1 Sequential Write PerformanceThe goal of our sequential write performance test

was to measure the maximum throughput that can beexpected under certain conditions. When a largeamount of data is written sequentially, our driverinvalidates blocks in each segment sequentially. Thedriver therefore needs no copying for cleaning asegment and the maximum write performance can beobtained.

Figure 5 shows the sequential write throughput asa function of cumulative data written, and Table 4summarizes the results. The results were obtained byfirst writing a certain amount of data and thenrepeatedly overwriting that initial data. The curvesshow results based on different initial data: 4.2 Mbytes(30% of the file system capacity), 8.4 Mbytes (60%),and 12.6 Mbytes (90%). The greedy policy was usedfor cleaning.

The results obtained with the 90% initial data loadwere unexpected. Although the data were overwrittensequentially, many blocks were copied for cleaning.This copying was a result of the effect of the cleaningthreshold described earlier. Since the live data counts

Flash memory Intel 28F008SA (1 Mbyte/chip)Banks 8Segments 64 (8 segments/bank)Segment size 256 KbytesData blocks 32256 (504 blocks/segment)Erase cycle 1 sec.

bandwidth 252 Kbytes/sec.Write bandwidth 400 Kbytes/sec.Read bandwidth 4 Mbytes/sec.

Flash Memory System

CPU IDT R3081 (R3000 compatible)Cache Instruction 16 Kbytes

Data 4 KbytesMemory bandwidth

Instruction read 20 Mbytes/sec.Data read 7 Mbytes/sec.Data write 5 Mbytes/sec.

CPU and Main Memory System

Table 3. Test-platform specifications.

†The actual hardware had 128 segments (16 segments/block), but inthe work reported here we used only half the segments of eachbank.

more than 53 segments for the 90% data, the cleaningthreshold was kept near 410 invalid blocks in asegment throughout the test. Consequently, the cleanercopied an average of 64 blocks per erasure andlowered the write throughput.

3.2 Random Write PerformanceThe random write performance test evaluated the

worst case for our driver. When a randomly selectedportion of a large amount of data is overwritten, all thesegments are invalidated equally. If the invalidationtakes place unevenly (e.g., sequentially), somesegments are heavily invalidated, and thus can becleaned with a small amount of copying. The eveninvalidation caused by the random update, however,results in there being less chance to clean segmentsthat are particularly highly invalidated. Therefore, thecleaning cost approaches a constant value for allsegments.

For our driver, the copying cost is expected to bea function of the ratio of used space to free space in thefile system. As new data are written to the freesegments, the used segments are invalidated evenly.The free segments are eventually exhausted and thecleaner starts cleaning. Consequently, the ratio of validto invalid blocks of each segment becomes that of theratio of used to free space of the file system.

Figure 6 and Table 5 show the results of therandom write test. These results were obtained bywriting a 4Kbyte data block to a randomly selectedposition of various amounts of initial data: again, 4.2,

Figure 5. Sequential write performance.

0

50

100

150

200

250

300

0 50 100 150 200

Writ

e Th

roug

hput

(KBy

te/s

)

Cumulative MBytes Written

/usr/motoG/tmp/ha1_10_20_94/cum-g-s-ni.eps

30%60%90%

Writ

e th

roug

hput

(KBy

tes/s

)

Cumulative MBytes written

Initialdata

Averagethroughput(Kbytes/s)

30% 231 749 0 19260% 230 766 0 19290% 199 889 57 266 192

Numberof erasedsegments

Numberof copied

blocks

Total data written

(Mbytes)

Table 4. Summary of sequential write performance.


8.4, and 12.6 Mbytes. And again the greedy policy wasused for cleaning.

3.3 Hot-and-Cold Write Performance

This test evaluated the performance cases wherewrite accesses exhibited certain amounts of thelocality of reference. In such cases, we can expect thecost-benefit policy to force the segments into abimodal distribution where most of the segments arenearly full of valid blocks and a few are nearly full ofinvalid blocks [3]. Consequently, the cost-benefitpolicy would result in a low copying overhead. Wewill see that our first results did not match ourexpectations; we will then analyze why this anomalyoccurred and what measures we took to address it.

Figure 7 and Table 6 show the results of the test.640-116 means that 60% of the write accesses go toone-eighth of the initial data and other 40% go toanother one-eighth. The other three-fourths of the dataare left unmodified. With this distribution we intend tosimulate meta data write activity on an actual filesystem. The ratio of writes is based on the resultsreported in [5], which found that 67-78% of writes areto meta data blocks. In all the tests we conducted, anactual write position in a selected portion of the initialdata was decided randomly, the size of the initial datawas 12.6 Mbytes (90%), and all writes were done in4Kbyte units.

Figure 6. Random write performance.

0

50

100

150

200

250

300

0 50 100 150 200

Writ

e Th

roug

hput

(KBy

te/s

)


/usr/motoG/tmp/ha1_10_20_94/cum-g-r-ni.eps

30%60%90%

Writ

e th

roug

hput

(KBy

tes/s

)


30% 222 801 26 383 19260% 147 1066 155 856 19290% 40 2634 938 294 192

Initialdata



Numberof copied

blocks

Total data written

(Mbytes)

Table 5. Summary of random write performance.

3.4 Separate Segment for CleaningThe initial results obtained in the hot-and-cold

write test were far worse than we had expected. Thewrite throughput of the 640-116 test was nearly thesame as that of the random test using the greedypolicy. Furthermore, the greedy policy worked betterthan the cost-benefit policy for the 640-116 test.

Figure 8 shows the distribution of segmentutilization after the 640-116 test. In the figure, we canobserve a weak bimodal distribution of segmentutilization. Since the 60% of the data was leftunmodified, more fully valid segments should bepresent.

We traced the blocks that the cost-benefit policyonce judged as cold in the test, and Figure 9 shows thedistribution of the cold and the not-cold blocks in thesegments after executing the 640-116 test. The data in

Figure 7. Hot-and-cold write performance.

30

35

40

45

50

55

60

0 50 100 150 200

Writ

e Th

roug

hput

(KBy

te/s

)


/usr/motoG/tmp/hot-cold-merged/cum.eps

640-116, Greedy

640-116, Cost-Benefit

Writ

e th

roug

hput

(KBy

tes/s

)


Nomuberof copied

blocks51 2617 925 193

43 3085 1 161 090

Initial data: 90% (12.6 Mbytes), Total data written: 192 Mbytes

TestAverage

throughput(Kbytes/s)

Nomuberof erasedsegments

Cleaningpolicy

640-116GreedyCost-Benefit

Table 6. Summary of hot-and-cold write performance.

Figure 8. Segment utilization distribution after the 640-116 test.

0

2

4

6

8

10

12

14

0 0.2 0.4 0.6 0.8 1

Num

ber o

f Seg

men

ts

Segment Utilization

util.90-2u-640_116-i-off-f.eps

Num

ber o

f seg

men

ts

Segment Utilization


this figure was obtained by marking a block as “cold”when the segment to which the block belongs waschosen to be cleaned and its utilization was less thanthe average utilization in the file system. We can seethat some segments contain both cold and not-coldblocks. Furthermore, the number of cold blocks ismuch smaller than expected: since three-fourths of the12.6 Mbytes of initial data were left unmodified, wewould expect, in the best case, about 19 000 coldblocks (i.e., about 38 cold segments). In the test,however, the actual number of cold blocks was 2579.

The reason we determined for the above results isthat the driver uses one segment for both the datawrites and the cleaning operations; the valuable,potentially cold blocks are mixed with data beingwritten to the segment. The number of cold blockstherefore does not increase over time.

To address this problem, we modified the driverso that the driver uses two segments: one for cleaningcold segments and one for writing the data andcleaning the not-cold segments. Table 7 summarizesthe results of 640-116 tests on both the modified andthe original drivers. The effect of the separate cleaningsegment becomes notable as the initial utilizationgrows, and the write throughput was improved morethan 40% for the 90% initial data. Figure 10 shows the

Figure 9. Distribution of cold and not-cold blocks after the 640-116 test.

0.0 0.25 0.5 0.75 1.00.0

0.25

0.5

0.75

1.0

0

5

10

15

20

25

Ratio ofnot-coldblocks

Ratio of cold blocks

Num

ber o

f seg

men

tsNumber

of copiedblocks

No 241 742 0Yes 239 744 0

No 197 888 63 627Yes 198 832 33 997

No 135 1195 214 894Yes 143 883 57 205

No 81 1855 544 027Yes 127 1089 157 922

No 43 3085 1 161 090Yes 60 2218 723 582

Total data written: 192 Mbytes

Initialdata



Separatesegment

30%

60%

70%

80%

90%

Table 7. Summary of 640-116 tests using the separate cleaning segment.

distribution of cold blocks after the 640-116 test usingthe modified driver. Many cold segments areobserved.

3.5 Andrew BenchmarkTable 8 lists the results of the Andrew

benchmark [6] for MFS and for our prototype. Theresults were obtained by repeating the benchmark run60 times. The output data files and directories of eachrun were stored in a directory, and to limit the filesystem usage the oldest directory was removed beforeeach run after 14 contiguous runs for the 52-56% test,after 24 for the 92-96% test, and after 9 for the MFStest. Note that, as pointed out in [4], phases 3 and 4performed no I/O because all the data access werecached by the higher-level buffer and the inode caches.

The benchmark consists of many read operationsand leaves a total of only about 560 Kbytes of data foreach run. As a result, there are many chances to cleansegments without disturbing data write operations.Therefore, our prototype shows performance nearlyequivalent to that of MFS. We expect that similaraccess patterns often appear in a personal computingenvironment. Note that the cleaner erased 903

Figure 10. Distribution of cold and not-cold blocks after the 640-116 test using the separate cleaning segment.

0.0 0.25 0.5 0.75 1.00.0

0.25

0.5

0.75

1.0

0

5

10

15

20

25

Ratio of cold blocks

Ratio ofnot-coldblockN

umbe

r of s

egm

ents

MFS52-56% 92-96%

Prototype

Phase 1 1.3 2.0 2.9Phase 2 8.0 9.5 11.5Phase 3 13.1 13.5 13.5Phase 4 16.9 16.9 17.1Phase 5 80.6 81.8 84.0

Total 119.9 123.7 129.0

Averageelapsed

timefor

eachrun

Number of writtenblocks for dataNumber of copiedblocksNumber of erasedsegments

251 818 233 702

75 227 255 020

578 903

Table 8. Andrew Benchmark results.


segments for the 92-96% test; under the same load (15segments in 2 minutes), our prototype will surviveabout 850 000 minutes (590 days).

4. Related WorkLogging has been widely used in certain kinds of

devices; in particular, in Write-Once Read-Many(WORM) optical disk drives. WORM media areespecially suitable for logging because of theirappend-only writing. OFC [8] is a WORM-based filesystem that supports a standard UNIX file systemtransparently. Although its data structures differ fromthose of our prototype, our prototype's block-addresstranslation scheme is very similar to that of OFC. OFCis self-contained in that it stores all data structures ona WORM medium and needs no read-write mediumsuch as a HDD. To get around the large memoryrequirement, it manages its own cache to provideefficient access to the structure. Our prototype,however, needs improvement with regard to itsmemory requirement (about 260 Kbytes for a16-Mbyte flash memory system).

LFS [3] uses the logging concept for HDDs. Ourprototype is similar to LFS in many aspects, such assegment, segment summary, and segment cleaner, butLFS does not use block-address translation. LFSincorporates the FFS index structure into the log sothat data retrieval can be made in the same fashion asin the FFS. That is, each inode contains pointers thatdirectly point to data blocks. Our prototype, on theother hand, keeps a log of physical block modification.

LFS gathers as many data blocks as possible to bewritten in order to maximize the throughput of writeoperations by minimizing seek operations. Since flashmemory is free from seek penalty, maximizing thewrite size does not necessarily improve performance.

The paper on BSD-LFS [4] reports that the effectof the cleaner is significant when data blocks areupdated randomly. Under these conditions, eachsegment tends to contain fewer invalid data blocks andthe cleaner's copying overhead accounts for more than60% of the total writes. With our prototype, thisoverhead accounts for about 70% on the 90%-utilizedfile system.

Since flash memory offers a limited number ofwrite/erase cycles on its memory cell, our driverrequires the block translation mechanism. LogicalDisk (LD) [9] uses the same technique to make a disk-based file system log-structured transparently.Although the underlying storage media and goals aredifferent, both the driver and LD function similarly.LD does, though, provides one unique abstractinterface called block lists. The block lists enable a file

system module to specify logically related blocks suchas an inode and its indirect blocks. Such an interfacemight be useful for our driver by enabling it to clusterhot and cold data.

Douglis et al. [10] have examined three devicesfrom the viewpoint of mobile computing: a HDD, aflash disk, and a flash memory. Their simulationresults show that the flash memory can use 90% lessenergy than a disk-based file system and respond up totwo orders of magnitude faster for read but up to anorder of magnitude slower for write. They also foundthat, at 90% utilization or above, a flash memoryerasure unit that is much larger than the file systemblock size will result in unnecessary copying forcleaning and will degrade performance.

The flash-memory-based storage systemeNVy [11] tries to provide high performance in atransaction-type application area. It consists of a largeamount of flash memory, a small amount of battery-backed SRAM for write buffering, a large-bandwidthparallel data path between them, and a controller forpage mapping and cleaning. In addition to thehardware support, it uses a combination of twocleaning policies, FIFO and locality gathering, inorder to minimize the cleaning costs for both uniformand hot-and-cold access distribution. Simulationresults show that at a utilization of 80% it can handle30 000 transactions per second while spending 30%processing time for cleaning.

Microsoft Flash File System (MFFS) [2] providesMS-DOS-compatible file system functionality with aflash memory card. It uses data regions of variable sizerather than data blocks of fixed length. Files in MFFSare chained together by using address pointers locatedwithin the directory and file entries. Douglis et al. [10]observed that MFFS write throughput decreasedsignificantly with more cumulative data and with morestorage consumed.

SunDisk manufactures a flash disk card that has asmall erasure unit, 576 bytes [12]. Each unit takes lesstime to be erased than does Intel’s 16Mbit flashmemory. The size enables the card to replace a HDDdirectly. The driver of the card erases data blocksbefore writing new data into them. Although this eraseoperation reduces the effective write performance, theflash disk card shows stable performance under highutilization because there is no need to copy livedata [10]. In a UNIX environment with FFS, simplyreplacing the HDD with the flash disk would result inunexpected short life because FFS meta data such asinodes are located at fixed blocks and are updatedmore often than user data blocks. The flash disk cardmight perform well in the UNIX environment if aproper wear-leveling mechanism were provided.


5. Conclusion

Our prototype shows that it is possible toimplement a flash-memory-based file system forUNIX. The benchmark results shows that the proposedsystem avoids many of the problems expected to resultfrom flash memory's overwrite incapability.

The device driver approach makes it easy toimplement this prototype system by using the existingFFS module. But because the FFS is designed for usewith HDD storage, this prototype needs to use aportion of the underlying flash memory to hold datastructures tuned for a HDD. Furthermore, theseparation of the device driver from the file systemmodule makes the prototype system managementdifficult and inefficient. For example, there is no wayfor the driver to know whether or not a block isactually invalid until the FFS module requests a writeon the block—even if the file for which the block wasallocated had been removed 15 minutes before. A filesystem module should therefore be dedicated to flashmemory.

AcknowledgmentsWe thank the USENIX anonymous referees for

their comments, Douglas Orr for valuable suggestionsand comments on the drafts, Fred Douglis for makinghis draft available to us, and Satyanarayanan-san andthe Information Technology Center, Carnegie-MellonUniversity for providing us the Andrew Benchmark.

Microsoft and MS-DOS are registered trademarksof Microsoft Corporation.

References[1] Advanced Micro Devices, Inc., “Am29F040

Datasheet”, 1993.[2] Flash Memory, Intel Corporation, 1994.[3] M. Rosenblum and J. K. Ousterhout, “The Design

and Implementation of a Log-Structured FileSystem”, ACM Transactions on ComputerSystems, Vol. 10, No. 1, 1992.

[4] M. Seltzer, K. Bostic, M. K. McKusick, and C.Staelin, “An Implementation of a Log-StructuredFile System for UNIX”, Proc. ’93 WinterUSENIX, 1993.

[5] C. Ruemmler and J. Wilkes, “UNIX disk accesspatterns”, Proc. ’93 Winter USENIX, 1993.

[6] J. H. Howard, et al., “Scale and Performance in aDistributed File System”, ACM Transactions onComputer Systems, Vol. 6, No. 1, 1988.

[7] M. K. McKusick, M. J. Karels, and K. Bostic, “APageable Memory Based Filesystem”, Proc. ’90Summer USENIX, 1990.

[8] T. Laskodi, B. Eifrig, and J. Gait, “A UNIX FileSystem for a Write-Once Optical Disk”, Proc. ’88Summer USENIX, 1988.

[9] W. de Jonge, M. F. Kaashoek, and W. C. Hsieh,“Logical Disk: A Simple New Approach toImproving File System Performance”, TechnicalReport MIT/LCS/TR-566, MassachusettsInstitute of Technology, 1993.

[10] F. Douglis, R. Cáceres, F. Kaashoek, K. Li, B.Marsh, and J. A. Tauber, “Storage Alternativesfor Mobile Computers”, Proc. 1st Symposium onOperating Systems Design and Implementation,1994.

[11] M. Wu and W. Zwaenepoel, “eNVy: ANon-Volatile, Main Memory Storage System”,Proc. 6th International Conference onArchitectural Support for ProgrammingLanguages and Operating Systems, 1994.

[12] “Operating system now has flash EEPROMmanagement software for external storagedevices” (in Japanese), Nikkei Electronics, No.605, 1994.

Author InformationAtsuo Kawaguchi is a research scientist at the

Advanced Research Laboratory, Hitachi, Ltd. Hisinterests include file systems, memory managementsystem, and microprocessor design. He received B.S.,M.S., and Ph.D. degrees from Osaka University. Hecan be reached at [email protected].

Shingo Nishioka is a research scientist at theAdvanced Research Laboratory, Hitachi, Ltd. Hisinterests include programming languages andoperating systems. He received B.S., M.S., and Ph.D.degrees from Osaka University. He can be reached [email protected].

Hiroshi Motoda has been with Hitachi since 1967and is currently a senior chief research scientist at theAdvanced Research Laboratory and heads the AIgroup. His current research includes machine learning,knowledge acquisition, visual reasoning, informationfiltering, intelligent user interfaces, and AI-orientedcomputer architectures. He received his B.S., M.S.,and Ph.D. degrees from the University of Tokyo. Hewas on the board of trustees of the Japan Society ofSoftware Science and Technology and of the JapaneseSociety for Artificial Intelligence, and he was on theeditorial board of Knowledge Acquisition and IEEEExpert. He can be reached [email protected].

All the authors can be reached atAdvanced Research Laboratory, Hitachi, Ltd.Hatoyama, Saitama, 350-03 Japan.

Documents

A Flash-Memory Based File System