Upload
ngothu
View
222
Download
0
Embed Size (px)
Citation preview
Paruj Ratanaworabhan,
Ben Livshits, and Ben Zorn
JSMeter: Characterizing the Behavior of JavaScript Web Applications
1
In collaboration with
David Simmons, Corneliu Barsan, and Allen Wirfs-Brock
Why Measure JavaScript?
• Standardized, de facto language for the web
– Support in every browser, much existing code
• Major Microsoft investment
– Web Application Companion based on Script#
• Benchmarks are driving browser marketing: “browser wars”
– Are benchmarks representative?
– Limited understanding of its behavior in real sites
• Who cares?
– Users, web site designers, JavaScript engine developers
2
Artificial Benchmarks versus Real World Sites
4
Maps Maps
7 V8 programs:
• richards
• deltablue
• crypto
• raytrace
• earley-boyer
• regexp
• splay
8 SunSpiderprograms:
• 3-draytrace
• access-nbody
• bitops-nsieve
• controlflow
• crypto-aes
• date-xparb
• math-cordic
• string-tagcloud
JSMeter
Goals of this Talk
• Show behavior of JavaScript benchmarks versus real sites
• Consider how benchmarks can mislead design decisions
11 real sites
5
Instrumentation Framework
\ie\jscript\*.cpp
Source-level instrumentation
custom jscript.dll
custom trace fileswebsite visits
Offline analyzers
custom trace files
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Constant
Other Str Ops
Concat Op
Visiting the Real Sites
• Attempted to use each site in “normal” way:
6
amazon Search a book, add to shopping cart, sign in, and sign out
bing Type in a search query and also look for images and news
bingmap Search for a direction from one city to another
cnn Read front page news
ebay Search for a notebook, bid, sing in, and sign out
economist Read front page news, view comments
facebook Log in, visit a friend pages, browse through photos and comments
gmail Sign in, check inbox, delete a mail, and sign out
google Type in a search query and also look for images and news
googlemap Search for a direction from one city to another
hotmail Sign in, check inbox, delete a mail, and sign out
Code Behavior
Code
Events
JavaScript
Objects
8
• Function size
• Instructions/call
• Code locality
• Instruction mix
Total Bytes of JavaScript Source
9
0
500
1000
1500
2000
2500
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ab
lue
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp-dna
str
ing-ta
gclo
ud
Sourc
e s
ize (kilob
yte
s)
Real Sites V8 SunSpider
Code|Objects|Events
Static Unique Functions Executed
11
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ablu
e
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp-dna
str
ing-ta
gclo
ud
Uniq
ue F
uncti
ons E
xecute
d
Real Sites V8 SunSpider
Code|Objects|Events
Total Bytecodes Executed
12
0
5000
10000
15000
20000
25000
30000
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ab
lue
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp
-dna
str
ing-ta
gclo
ud
Tota
l Byte
codes
Excute
d(t
housands)
90395
Real Sites V8 SunSpider
Code|Objects|Events
Bytecodes / Call
14
0.00
50.00
100.00
150.00
200.00
250.00
300.00
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ab
lue
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp
-dna
str
ing-ta
gclo
ud
Byte
codes /
Call
function(a,b) {var i=0,elem,pos=a.length;if(D.browser.msie) {
while(elem=b[i++])if(elem.nodeType!=8)
a[pos++]=elem;} else
while(elem=b[i++])a[pos++]=elem;
return a}
Real Sites V8 SunSpider
Code|Objects|Events
Fraction of Code Executed
15
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
am
azon
bin
gbin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
sdelt
ab
lue
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
icre
gexp-dna
str
ing-ta
gclo
ud
Fra
cti
on o
f C
od
e E
xecute
d
Most code
not executed
Real Sites V8 SunSpider
Code|Objects|Events
Hot Function Distribution
16
Real Sites V8 Benchmarks
80% of time in < 10 functions80% of time in 100+ functions
80% 80%
Code|Objects|Events
Opcode Distribution
17Green = SunSpider | Blue= Real Web Apps | Red = V8
Real Apps
Outliers
Code|Objects|Events
Object Allocation Behavior
Code
Events
JavaScript
Objects
18
• Allocation by types
• Lifetime distribution
• Live heap composition
Total Bytes Allocated
19
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
am
azon
bin
g
bin
gm
ap
cnn
ebay
econom
ist
gm
ail
map
hotm
ail
richard
s
delt
ab
lue
cry
pto
raytr
ace
earl
ey
regexp
spla
y
3d-ra
ytr
ace
access-nbody
bit
ops-nsie
ve
contr
olf
low
cry
pto
-aes
date
-xparb
math
-cord
ic
regexp
-dna
str
ing-ta
gclo
ud
Tota
l heap
data
(kilobyte
s)
Real Sites V8 SunSpider
Code|Objects|Events
Heap Data by Type
20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
amaz
on
bin
gb
ingm
ap cnn
ebay
eco
no
mis
tfa
ceb
oo
kgm
ail
goo
gle
goo
glem
aph
otm
ail
rich
ard
sd
elta
blu
ecr
ypto
rayt
race
earl
eyre
gexp
spla
yV
8 a
ggra
gate
3d
-ray
trac
eac
cess
-nb
od
yb
ito
ps-
nsi
eve
con
tro
lflo
wcr
ypto
-aes
dat
e-xp
arb
mat
h-c
ord
icre
gexp
-dn
ast
rin
g-ta
gclo
ud
aggr
egat
e
He
ap D
ata
by
Typ
e
Objects
Enumarator
RegExp
Math
Error
Date
Native Func
String
Arrays
Script Func
Real Sites V8 SunSpider
Many functions
Rest are strings
Few benchmarks
allocate much data
Code|Objects|Events
Live Heap Over Time (gmail)
22
Functions grow
steadily
Code|Objects|Events
GC reduces
size of heap
Objects grow
steadily too
Live Heap over Time (ebay)
23
Heaps
repeatedly
created,
discarded
Heap contains
mostly functions
Code|Objects|Events
Heap drops to 0
on page load
Object Lifetimes (V8)
26
earley splay
Code|Objects|Events
Lives forever
Dies
immediately
Lives until
program ends
Object Lifetimes (gmail)
27
Strings are
shortest lived
Functions are
longest lived
Many objects
live a major
fraction of
page lifetime
This is as old
as an object
can be
Code|Objects|Events
Event Handlers in JavaScript
Code
Events
JavaScript
Objects
28
• Number of events• Sizes of handlers
Event-driven Programming Model
• Single-threaded, non-preemptive event handlers
• Example of some standard event-handler attributes:
– onabort, onclick, and onmouseover
• Very different from batch processing of benchmarks
• Handler responsiveness critical to user experience
29
Code|Objects|Events
Total Events Handled
30
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000T
ota
l Events
Hand
led
Real Sites V8
Almost no events
Code|Objects|Events
Median Bytecodes / Event Handled
32
0
50
100
150
200
250
300
350
400
450
500M
edia
n B
yte
codes /
Event
Handle
d
Code|Objects|Events
Maximum Bytecodes / Event Handled
33
0
200,000
400,000
600,000
800,000
1,000,000
1,200,000M
axim
um
Byte
codes /
Event
Handle
d
Code|Objects|Events
A Simple Experiment
• Observation
– Real web apps have lots of code (much of it cold)
– Benchmarks do not
• Question: What happens if the benchmarks have more code?
– We added extra, unused to code to 6 SunSpiderbenchmarks
– We measured the impact on the benchmark performance
36
Performance Impact of Cold Code
37
0
100
200
300
400
500
600
700
800
900
Tim
e (
mse
c)
0K
200K
400K
800K
1M
2M
0
100
200
300
400
500
600
700
800
900
Tim
e (
mse
c)
0K
200K
400K
800K
1M
2M
Chrome IE 8
Cold code has
non-uniform impact
on execution time
Without cold code,
Chrome12x faster than IE8With 2M cold code,
Chrome 4.7x faster than IE8
Which comparison is more meaningful?
Cold code makes
SunSpider on Chrome
up to 4.5x slower
Impact of Benchmarks
• What gets emphasis
– Making tight loops fast
– Optimizing small amounts of code
• Important issues ignored
– Garbage collection (especially of strings)
– Managing large amounts of code
– Optimizing event handling
– Considering JavaScript context between page loads
38
Conclusions
• JSmeter is an instrumentation framework
– Used to measure and compare JavaScript applications
– High-level views of behavior promote understanding
• Behavior in benchmarks significantly differs from real sites
– Misleads designers, skews implementations
• Next steps
– Develop and promote better benchmarks
– Design and evaluate better JavaScript runtimes
– Promote better performance tools for JavaScript developers
39
Additional Resources
• Project– http://research.microsoft.com/en-us/projects/jsmeter/
• Papers– "JSMeter: Measuring JavaScript Behavior in the Wild", Paruj Ratanaworabhan,
Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2010-8, January 2010 (17 pages)
– "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages)
– “An Analysis of the Dynamic Behavior of JavaScript Programs”, Gregor Richards, Sylvain Lebresne, Brian Burg, Jan Vitek (to appear in PLDI 2010)
40