Embed Size (px)
Citation preview
Replacing RabbitMQ with MongoDB
David Mytton
Woop Japan!
www.flickr.com/photos/triplexpresso/496995086/
Queuing: Uses
www.flickr.com/photos/triplexpresso/496995086/
Queuing: Uses
• Background processing
www.flickr.com/photos/triplexpresso/496995086/
Queuing: Uses
• Background processing
• Sending notifications
www.flickr.com/photos/triplexpresso/496995086/
Queuing: Uses
• Background processing
• Sending notifications
• Event streaming
Queuing: Features
Queuing: Features
• Consumers
Queuing: Features
• Atomic
• Consumers
Queuing: Features
• Speed
• Atomic
• Consumers
Queuing: Features
• Speed
• Atomic
• Consumers
• GC
Queuing: Features
•Consumers
Queuing: Features
•Consumers
MongoDB RabbitMQ
Mongo Wire Protocol AMQP
en.wikipedia.org/wiki/State_of_matter
Queuing: Features
•Atomic
en.wikipedia.org/wiki/State_of_matter
Queuing: Features
•Atomic
MongoDB RabbitMQ
findAndModify consume/ack
Queuing: Features
•Speed
Queuing: Features
•GC
Queuing: Features
•GC
MongoDB RabbitMQ
☹ consume/ack
It’s a little different.
Implementation
• Consumers
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
{ query: { hats: 5 } }
query: filter (WHERE)
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
{ sort: { added: -1 } }
sort: selects the first one on multi-match
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
{ update: { $set: {inProg: true, start: new Date()} } }
update: modifier object
Implementation
• Consumers
db.runCommand( { findAndModify : <collection>, <options> } )
remove: true = deletes on returnnew: true = returns modified objectfields: return specific fieldsupsert: true = create object if !exists()
Implementation
• Consumers
Implementation
• Consumers
def run(self): import time, sched sc = sched.scheduler(time.time, time.sleep) self.poll(sc) sc.run()
def poll(self, sc): doStuff() sc.enter(10, 1, self.poll, (sc,))
Implementation
• Consumers
Implementation
• Consumers
from daemon import Daemon class pantalaimon(Daemon): def run(self): # Do stuff
pineMarten = pantalaimon('/path/to/pid.pid') pineMarten.start()
https://github.com/boxedice/python-daemon
Implementation
• Consumers
https://github.com/boxedice/python-daemon
from daemon import Daemon
class pantalaimon(Daemon): def run(self): import time, sched sc = sched.scheduler(time.time, time.sleep) self.poll(sc) sc.run()
def poll(self, sc): doStuff() sc.enter(10, 1, self.poll, (sc,))
Implementation
• GC
Implementation
• GC
now = datetime.datetime.now()difference = datetime.timedelta(seconds=10)timeout = now - difference
queue.find({'inProg' : True, 'start' : {'$lte' : timeout} })
Fault tolerance
Replica sets
Fault tolerance
Replica sets
Basic automatic failover
Fault tolerance
Replica sets
Basic automatic failover
Writing to n slaves
Fault tolerance
Replica sets
Basic automatic failover
Writing to n slaves
Data centre awareness
Scaling
Capped Collections
Scaling
Capped Collections
Extremely fast
Scaling
Capped Collections
Maintain insertion order (FIFO)
Extremely fast
Scaling
Capped Collections
Don’t have to remove()
Maintain insertion order (FIFO)
Extremely fast
Scaling
Sharding
Scaling
Sharding
findAndModify() locks
Scaling
Sharding
Scaling writes across machines
findAndModify() locks
Scaling
Sharding
Scaling
Sharding
Not with capped collections
Problems / Mongo quirks
Problems / Mongo quirks
Meta
Problems / Mongo quirks
Meta
Durability
Problems / Mongo quirks
Meta
Durability
Global lock
Stick with RabbitMQ?
Stick with RabbitMQ?
QoS
Stick with RabbitMQ?
AMQP
QoS
AMQP
Stick with RabbitMQ?
Throttling
QoS
