Graybox NFS Caching Proxy

By:

Paul Cychosz and

Garrett Kolpin

• First we need to know a little about how the “graybox” works.

• Inference by observation

• No server or client-side modifications necessary

What are Graybox Techniques?

• Figure out what the client is caching

• Maintain low overlap between client and proxy caches

Our Problem

• Treat the client as a graybox

• Monitor NFS traffic

• Infer from NFS traffic the contents of client cache

Our Solution

Cache overlap decreased by 5% to 50% when using graybox techniques

Conclusions

• Approach

• Results

• Conclusions

Outline

Approach

NFS specifics: Implemented in kernel as another FS type

Replaces “file_operations” with its own functions for read, write, open, etc.

Uses VFS abstraction

VFS

NFS kernel code

RPC / XDR

TCP / UDP

retrans., data wrapping

Approach

Client cache / Replacement

• Caches on page level • Not much NFS specific code – Uses global Linux page cache

• Not a static size, no upper-bound

• Timestamps: “jiffies”, per page.

Approach

Reads typically > ~85% of traffic

NFS / RPC “read” structure

• RPC Packets sent out: access to dir, getattr to file, access to file, then read, read, read, …

• Client use of getattr to revalidate cached copies

Exploit this, proxy explicitly look for these to determine client cache contents

Approach

ProxySees:

File handle

Offset

size

--becomes our “cache element”

Store in hash table

Cache on reads, run our cache element replacement policy on getattr (proof of cached client copy)

Approach

Read system call

Cache lookup

Revalidate data

Read setup

OR

Replacement(possible force-out)

Lookup

Return data Add

OR

NFS Client Proxy

hit

miss

hit

miss

getattr

read

Data

Forward req. to server

Typical read()

• Disable swap space

• Use mincore() to view pages that client has cached

• Use “balloon” program to keep client cache contents manageable

Viewing Client Cache Contents

Test Environment:

Approach

Logistics: Initial setup, client/server/proxy all on same PC – doesn’t work

Client cache size changes

-solved: compare percentages, normalize

Repeated RPC calls not idempotent

Difficulties

Not a significant problem, proxy efficiency degrades linearly with NFS performance if network traffic is bad

Approach

Ambiguities:

Different FS activities DON’T generate unique RPC patterns

getattr used for a few things besides reads

-- Null RPC packets

Cannot tell exact blocks client caching, only which files

Need second, local replacement policy on proxy!

Difficulties

Approach

Global Policies

• LRU

• MRU

Local Policies

• Forcing entire file

• MRU

Cache Replacement

• Approach

• Results

• Conclusions

Outline

• Control Experiments– Global LRU, no local force-out– Global MRU, no local force-out

• Combinations– Global LRU, whole file force-out– Global LRU, block-level local MRU– Global MRU, whole file force-out– Global MRU, block-level local MRU

What We Tested

• Used Postmark benchmark

• 12 files with sizes randomly selected between 4 and 50 kilobytes

• Gives us more of a “real-world” workload

How We Tested

LRU Global Replacement

0

20

40

60

80

100

120

NF WF MRU

Force-out Method

Per

cen

t

Proxy Cache Fill

Overlap

Results

MRU Global Replacement

0

20

40

60

80

100

120

NF WF MRU

Force-out Method

Per

cen

t

Proxy Cache Fill

Overlap

Results

All Combinations

0

20

40

60

80

100

120

LRU NF MRU NF LRU WF LRU MRU MRU WF MRU MRU

Replacement Combination

Per

cen

t

Proxy Cache Fill

Overlap

Results

• Approach

• Results

• Conclusions

Outline

• Decrease in overlap by between 5 and 50 percent as compared to control tests

• MRU global file replacement is good by itself

• MRU global policy with whole-file local policy gives highest cache fill and lowest overlap

Conclusions

Questions? Feedback?

?

?

?