Upload
aman-gupta
View
3.408
Download
0
Embed Size (px)
Citation preview
Debugging Rubywith MongoDB
Aman Gupta@tmm1
Ruby developers know...
Rubyis
fatboyke (flickr)
Ruby loves eating RAM
37prime (flickr)
ruby allocates memory from the OS
memory is broken up into slots
each slot holds one ruby object
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
if the freelist is empty, GC is run
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
if the freelist is empty, GC is run
GC finds non-reachable objects and adds them to the freelist
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
if the freelist is empty, GC is run
GC finds non-reachable objects and adds them to the freelist
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
if the freelist is still empty (all slots were in use)
if the freelist is empty, GC is run
GC finds non-reachable objects and adds them to the freelist
when you need an object, it’s pulled off the freelist
a linked list called the ‘freelist’ points to all the
empy slots on the ruby heap
if the freelist is still empty (all slots were in use)
another heap is allocated
all the slots on the new heap are added to the freelist
turns out,
Ruby’s GC is
also one of the
reasons it can be so
slowantphotos (flickr)
Matz’ Ruby Interpreter (MRI 1.8)has a...
john_lam (flickr)
Conservativelifeisaprayer (flickr)
Stopthe
Worldbenimoto (flickr)
Markand
Sweepmichaelgoodin (flickr)
Garbage Collector
kiksbalayon (flickr)
•conservative: the VM hands out raw pointers to ruby objects
•stop the world: no ruby code can execute during GC
•mark and sweep: mark all objects in use, sweep away unmarked objects
more objects=
longer GC
mckaysavage (flickr)
longer GC=
less time to run your ruby code
kgrocki (flickr)
fewer objects=
better performance
januskohl (flickr)
improve performance1. remove unnecessary object allocations
object allocations are not free
improve performance1. remove unnecessary object allocations
object allocations are not free
2. avoid leaked referencesnot really memory ‘leaks’
you’re holding a reference to an object you no longer need. GC sees the reference, so it keeps the object around
the GC follows
references recursively, so a reference
to classA will ‘leak’ all these objects
let’s build a debugger
• step 1: collect data
• list of all ruby objects in memory
• step 2: analyze data
• group by type
• group by file/line
• simple patch to ruby VM (300 lines of C)
• http://gist.github.com/73674
• simple text based output format
0x154750 @ -e:1 is OBJECT of type: T0x15476c @ -e:1 is HASH which has data0x154788 @ -e:1 is ARRAY of len: 00x1547c0 @ -e:1 is STRING (SHARED) len: 2 and val: hi0x1547dc @ -e:1 is STRING len: 1 and val: T0x154814 @ -e:1 is CLASS named: T inherits from Object0x154a98 @ -e:1 is STRING len: 2 and val: hi0x154b40 @ -e:1 is OBJECT of type: Range
version 1: collect data
version 1: analyze data$ wc -l /tmp/ruby.heap
1571529 /tmp/ruby.heap
version 1: analyze data
$ cat /tmp/ruby.heap | awk '{ print $3 }' | sort | uniq -c | sort -g | tail -1
236840 memcached/memcached.rb:316
$ wc -l /tmp/ruby.heap
1571529 /tmp/ruby.heap
version 1: analyze data
$ cat /tmp/ruby.heap | awk '{ print $3 }' | sort | uniq -c | sort -g | tail -1
236840 memcached/memcached.rb:316
$ grep "memcached.rb:316" /tmp/ruby.heap | awk '{ print $5 }' | sort | uniq -c | sort -g | tail -5
10948 ARRAY 20355 OBJECT 30744 DATA 64952 HASH 123290 STRING
$ wc -l /tmp/ruby.heap
1571529 /tmp/ruby.heap
version 1
• it works!
• but...
• must patch and rebuild ruby binary
• no information about references between objects
• limited analysis via shell scripting
• better data format
• simple: one line of text per object
• expressive: include all details about object contents and references
• easy to use: easy to generate from C code & easy to consume from various scripting languages
version 2 goals
equanimity (flickr)
version 2 is memprof• no patches to ruby necessary
• gem install memprof
• require ‘memprof’
• Memprof.dump_all(“/tmp/app.json”)
• C extension for MRI ruby VMhttp://github.com/ice799/memprof
• uses libyajl to dump out all ruby objects as json
{ "_id": "0x19c610",
"file": "file.rb", "line": 2,
"type": "string", "class": "0x1ba7f0", "class_name": "String",
"length": 10, "data": "helloworld"}
memory address of object
file and line where string was created
length and contentsof this string instance
address of the class “String”
stringsMemprof.dump{ "hello" + "world"}
floats and strings are separate ruby objects
{ "_id": "0x19c5c0",
"class": "0x1b0d18", "class_name": "Array",
"length": 4, "data": [ 1, ":b",
"0x19c750", "0x19c598" ]}
integers and symbols are stored in the array itself
arraysMemprof.dump{ [ 1, :b, 2.2, "d" ]}
hashes{ "_id": "0x19c598",
"type": "hash", "class": "0x1af170", "class_name": "Hash",
"default": null,
"length": 2, "data": [ [ ":a", 1 ], [ "0xc728", "0xc750" ] ]}
hash entries as key/value pairs
no default proc
Memprof.dump{ { :a => 1, "b" => 2.2 }}
classesMemprof.dump{ class Hello @@var=1 Const=2 def world() end end}
{ "_id": "0x19c408",
"type": "class", "name": "Hello", "super": "0x1bfa48", "super_name": "Object",
"ivars": { "@@var": 1, "Const": 2 }, "methods": { "world": "0x19c318" }}
class variables and constants are stored in the instance variable table
superclass object reference
references to method objects
version 2: memprof.coma web-based heap visualizer and leak analyzer
built on...
$ mongoimport -d memprof -c rails --file /tmp/app.json$ mongo memprof
let’s run some queries.
thaths (flickr)
how many objects?
how many objects?> db.rails.count()809816
• ruby scripts create a lot of objects
• usually not a problem, but...
• MRI has a naïve stop-the-world mark/sweep GC
• fewer objects = faster GC = better performance
brettlider (flickr)
what types of objects?
what types of objects?> db.rails.distinct(‘type’)
[‘array’, ‘bignum’, ‘class’, ‘float’, ‘hash’, ‘module’, ‘node’, ‘object’, ‘regexp’, ‘string’, ...]
mongodb: distinct• distinct(‘type’)
list of types of objects
• distinct(‘file’)list of source files
• distinct(‘class_name’)list of instance class names
• optionally filter first
• distinct(‘name’, {type:“class”})names of all defined classes
improve performancewith indexes
> db.rails.ensureIndex({‘type’:1})
> db.rails.ensureIndex( {‘file’:1}, {background:true})
mongodb: ensureIndex
• add an index on a field (if it doesn’t exist yet)
• improve performance of queries against common fields: type, class_name, super, file
• can index embedded field names
• ensureIndex(‘methods.add’)
• find({‘methods.add’:{$exists:true}})find classes that define the method add
darrenhester (flickr)
how many objs per type?
> db.rails.group({ initial: {count:0}, key: {type:true}, cond: {}, reduce: function(obj, out) { out.count++ }}).sort(function(a,b) { return a.count - b.count})
how many objs per type?
group on type
increment countfor each obj
sort results
[ ..., {type: ‘array’, count: 7621}, {type: ‘string’, count: 69139}, {type: ‘node’, count: 365285}]
• nodes represent ruby code
• stored like any other ruby object
• makes ruby completely dynamic
lots of nodes
how many objs per type?
mongodb: group
• cond: query to filter objects before grouping
• key: field(s) to group on
• initial: initial values for each group’s results
• reduce: aggregation function
mongodb: group• by type or class
• key: {type:1}• key: {class_name:1}
• by file & line• key: {file:1, line:1}
• by type in a specific file• cond: {file: “app.rb”},
key: {file:1, line:1}
• by length of strings in a specific file• cond: {file:“app.rb”,type:‘string’},
key: {length:1}
davestfu (flickr)
what subclasses String?
what subclasses String?> db.rails.find( {super_name:"String"}, {name:1})
{name: "ActiveSupport::SafeBuffer"}{name: "ActiveSupport::StringInquirer"}{name: "SQLite3::Blob"}{name: "ActiveModel::Name"}{name: "Arel::Attribute::Expressions"}{name: "ActiveSupport::JSON::Variable"}
select only name field
mongodb: find
• find({type:‘string’})all strings
• find({type:{$ne:‘string’}})everything except strings
• find({type:‘string’}, {data:1})only select string’s data field
http://body.builder.hu/imagebank/pictures/1088273777.jpg
the largest objects?
the largest objects?> db.rails.find( {type: {$in:['string','array','hash']} }, {type:1,length:1}).sort({length:-1}).limit(3) {type: "string", length: 2308}{type: "string", length: 1454}{type: "string", length: 1238}
mongodb: sort, limit/skip
• sort({length:-1,file:1})sort by length desc, file asc
• limit(10)first 10 results
• skip(10).limit(10)second 10 results
zoutedrop (flickr)
when were objs created?
when were objs created?• useful to look at objects over time
• each obj has a timestamp of when it was created
• find minimum time, call it start_time
• create buckets for every minute of execution sincestart
• place objects into buckets
when were objs created?> db.rails.mapReduce(function(){ var secs = this.time - start_time; var mins_since_start = secs % 60;
emit(mins_since_start, 1); }, function(key, vals){ for(var i=0,sum=0; i<vals.length; sum += vals[i++]); return sum; }, { scope: { start_time: db.rails.find().sort({time:1}).limit(1)[0].time } }){result:"tmp.mr_1272615772_3"}
start_time = min(time)
mongodb: mapReduce• arguments
•map: function that emits one or more key/value pairs given each object this
• reduce: function to return aggregate result, given key and list of values
• scope: global variables to set for funcs
• results
• stored in a temporary collection(tmp.mr_1272615772_3)
when were objs created?> db.tmp.mr_1272615772_3.count()12
script was running for 12 minutes
> db.tmp.mr_1272615772_3.find().sort({value:-1}).limit(1){_id: 8, value: 41231}
41k objects created 8 minutes after start
jeffsmallwood (flickr)
references to this object?
references to this object?ary = [“a”,”b”,”c”]
ary references “a”“b” referenced by ary
• ruby makes it easy to “leak” references
• an object will stay around until all references to it are gone
• more objects = longer GC = bad performance
• must find references to fix leaks
references to this object?• db.rails_refs.insert({
_id:"0xary", refs:["0xa","0xb","0xc"]})create references lookup table
• db.rails_refs.ensureIndex({refs:1})add ‘multikey’ index to refs array
• db.rails_refs.find({refs:“0xa”})efficiently lookup all objs holding a ref to 0xa
mongodb: multikeys
• indexes on array values create a ‘multikey’ index
• classic example: nested array of tags
• find({tags: “ruby”})find objs where obj.tags includes “ruby”
version 2: memprof.coma web-based heap visualizer and leak analyzer
memprof.coma web-based heap visualizer and leak analyzer
memprof.coma web-based heap visualizer and leak analyzer
memprof.coma web-based heap visualizer and leak analyzer
memprof.coma web-based heap visualizer and leak analyzer
memprof.coma web-based heap visualizer and leak analyzer
memprof.coma web-based heap visualizer and leak analyzer
memprof.coma web-based heap visualizer and leak analyzer
plugging a leak in rails3• in dev mode, rails3 is leaking 10mb per request
# in environment.rbrequire `gem which memprof/signal`.strip
let’s use memprof to find it!
plugging a leak in rails3
tell memprof to dump out the entire heap to json
$ memprof --pid <pid> --name <dump name> --key <api key>
send the app some requests so it leaks
$ ab -c 1 -n 30 http://localhost:3000/
2519 classes
30 copies of TestController
2519 classes
30 copies of TestController
mongo query for all TestController classes
details for one copy of TestController
find references to object
find references to object
find references to object
holding references to all controllers
“leak” is on line 178
• In development mode, Rails reloads all your application code on every request
• ActionView::Partials::PartialRenderer is caching partials used by each controller as an optimization
• But.. it ends up holding a reference to every single reloaded version of those controllers
• In development mode, Rails reloads all your application code on every request
• ActionView::Partials::PartialRenderer is caching partials used by each controller as an optimization
• But.. it ends up holding a reference to every single reloaded version of those controllers
Questions?
Aman Gupta@tmm1