Upload
denis-porter
View
227
Download
0
Embed Size (px)
DESCRIPTION
3 Beginning of the Story Flash Web Server on the SPECWeb99 benchmark yield very low score – : Standard Apache 400: Customized Apache 575: Best - Tux on comparable hardware However, Flash Web Server is known to be fast Motivation
Citation preview
Making the “Box” Transparent:
System Call Performance as a First-class Result
Yaoping Ruan, Vivek PaiPrinceton University
2
Outline
Motivation Design & implementation Case study More results
3
Beginning of the Story Flash Web Server on the SPECWeb99
benchmark yield very low score – 200 220: Standard Apache 400: Customized Apache 575: Best - Tux on comparable hardware
However, Flash Web Server is known to be fast
Motivation
4
Performance Debugging ofOS-Intensive ApplicationsWeb Servers (+ full SpecWeb99) High throughput Large working sets Dynamic content QoS requirements Workload scaling
How do we debug such a system?
CPU/networkdisk activity
multiple programslatency-sensitive
overhead-sensitive
Motivation
5
Guesswork, inaccuracy
OnlineMeasurement callsgetrusage()gettimeofday( )
High overhead > 40%DetailedCall graph profiling
gprof
CompletenessFastStatistical sampling
DCPI, Vtune, Oprofile
Guesswork, inaccuracy
OnlineMeasurement callsgetrusage()gettimeofday( )
High overhead > 40%DetailedCall graph profiling
gprof
CompletenessFastStatistical sampling
DCPI, Vtune, Oprofile
Current TradeoffsMotivation
6In-kernel measurement
Example of Current Tradeoffs
gettimeofday( )
Wall-clock time of an non-blocking system call
Motivation
7
Design Goals Correlate kernel information with
application-level information Low overhead on “useless” data High detail on useful data Allow application to control profiling
and programmatically react to information
Design
8
DeBox: Splitting Measurement Policy &
Mechanism Add profiling primitives into kernel Return feedback with each syscall
First-class result, like errno Allow programs to interactively profile
Profile by call site and invocation Pick which processes to profile Selectively store/discard/utilize data
Design
9
DeBox Architecture…res = read (fd, buffer, count)…
errnobuffer
read ( ) {Start profiling;…End profilingReturn;}
App
Kernel
DeBox Infoor
online
offline
Design
10
DeBox Data StructureDeBoxControl ( DeBoxInfo *resultBuf,
int maxSleeps, int maxTrace )
Trace Info
Sleep InfoBasic DeBox Info
01
…01
…2
Performance summary
Blocking information of the call
Full call trace & timing
Implementation
11
Sample Output: Copying a 10MB Mapped FileBasic
DeBox InfoPerSleepInf
o
…
CallTrace
# of CallTrace used0# of PerSleepInfo used2# of page faults989Call time (usec)3591064System call # (write( ))4
Basic DeBox Info
# processes on exit0# processes on entry1Line where blocked2727File where blockedkern/vfs_bio.cResource labelbiowrTime blocked (usec)723903# occurrences1270
PerSleepInfo[0]
fo_write( )3586472dofilewrite( )3590326fhold( )5holdfp( )25write( )3591064
CallTrace
……21210
…
(depth)
Implementation
12
In-kernel Implementation 600 lines of code in FreeBSD 4.6.2 Monitor trap handler
System call entry, exit, page fault, etc. Instrument scheduler
Time, location, and reason for blocking Identify dynamic resource contention
Full call path tracing + timings More detail than gprof’s call arc counts
Implementation
13
General DeBox OverheadsImplementation
14
Case Study: Using DeBox on the Flash Web server
Experimental setup Mostly 933MHz PIII 1GB memory Netgear GA621
gigabit NIC Ten PII 300MHz
clients
0
100
200
300
400
500
600
700
800
900
SPECWeb99 Results
orig
VM Patch
sendfile
Dataset size (GB): 0.12 0.75 1.38 2.01 2.64
Case Study
15
Example 1: Finding Blocking System
Callsopen()/190readlink()/
84read()/12stat()/6
unlink()/1…
Blocking system calls, resource blocked and their counts
inode - 28biord - 162inode - 84
biord - 3inode - 9inode - 6biord - 1
Case Study
16
Example 1 Results & Solution
Mostly metadata locking
Direct all metadata calls to name convert helpers
Pass open FDs using sendmsg( )
0
100
200
300
400
500
600
700
800
900
SPECWeb99 Results
orig
VM Patch
sendfile
FD Passing
Dataset size (GB): 0.12 0.75 1.38 2.01 2.64
Case Study
17
Example 2: Capturing Rare Anomaly
PathsFunctionX( ) {
…open( );… }
FunctionY( ) {open( );…open( ); }
When blocking happens:
abort( ); (or fork + abort)
Record call path
1. Which call caused the problem
2. Why this path is executed (App + kernel call trace)
Case Study
18
Example 2 Results & Solution
Cold cache miss path
Allow read helpers to open cache missed files
More benefit in latency
0
100
200
300
400
500
600
700
800
900
SPECWeb99 Results
orig
VM Patch
sendfile
FD Passing
FD Passing
Dataset size (GB): 0.12 0.75 1.38 2.01 2.64
Case Study
19
Example 3: Tracking Call History & Call
Trace
Call time of fork( ) as a function of invocation
Call trace indicates:
• File descriptors copy - fd_copy( )
• VM map entries copy - vm_copy( )
Case Study
20
Example 3 Results & Solution
Similar phenomenon on mmap( )
Fork helper eliminate mmap( )
0
100
200
300
400
500
600
700
800
900
SPECWeb99 Results
origVM Patch
sendfileFD Passing
Fork helperNo mmap()
Dataset size (GB): 0.12 0.75 1.38 2.01 2.64
Case Study
21
Sendfile Modifications(Now Adopted by
FreeBSD)
Cache pmap/sfbuf entries Return special error for cache misses Pack header + data into one packet
Case Study
22
SPECWeb99 Scores
0.12 0.75 1.38 2.01 2.64
0 100 200 300 400 500 600 700 800 900orig
sendfileFD PassingFork helper
No mmap()
VM Patch
New CGI InterfaceNew sendfile( )
Dataset size (GB)0.43 1.06 1.69 2.32 2.95
Case Study
23
New Flash Summary Application-level changes
FD passing helpers Move fork into helper process Eliminate mmap cache New CGI interface
Kernel sendfile changes Reduce pmap/TLB operation New flag to return if data missing Send fewer network packets for small files
Case Study
24
Throughput on SPECWeb Static Workload
0100200300400500600700
500 MB 1.5 GB 3.3 GBDataset Size
Thro
ughp
ut (
Mb/
s)
Apache Flash New-Flash
Other Results
25
Latencies on 3.3GB Static Workload
44X
47X
Mean latency improvement: 12X
Other Results
26
Throughput Portability (on Linux)
(Server: 3.0GHz P4, 1GB memory)
0200400600800
100012001400
500 MB 1.5 GB 3.3 GBDataset Size
Thro
ughp
ut (
Mb/
s)
Haboob Apache Flash New-Flash
Other Results
27
Latency Portability(3.3GB Static Workload)
3.7X
112X
Mean latency improvement: 3.6X
Other Results
28
Summary DeBox is effective on OS-intensive application
and complex workloads Low overhead on real workloads Fine detail on real bottleneck Flexibility for application programmers
Case study SPECWeb99 score quadrupled
Even with dataset 3x of physical memory Up to 36% throughput gain on static workload Up to 112x latency improvement Results are portable
30
SpecWeb99 Scores Standard Flash 200 Standard Apache 220 Apache + special module 420 Highest 1GB/1GHz score 575 Improved Flash 820 Flash + dynamic request module 1050
31
SpecWeb99 on Linux Standard Flash 600 Improved Flash 1000 Flash + dynamic request module 1350
3.0GHz P4 with 1GB memory
32
New Flash Architecture