MongoUK 2011 - Rplacing RabbitMQ with MongoDB

Replacing RabbitMQ with MongoDB

David Mytton

Woop Japan!

www.flickr.com/photos/triplexpresso/496995086/

Queuing: Uses

www.flickr.com/photos/triplexpresso/496995086/

Queuing: Uses

• Background processing

www.flickr.com/photos/triplexpresso/496995086/

Queuing: Uses

• Background processing

• Sending notifications

www.flickr.com/photos/triplexpresso/496995086/

Queuing: Uses

• Background processing

• Sending notifications

• Event streaming

Queuing: Features

Queuing: Features

• Consumers

Queuing: Features

• Atomic

• Consumers

Queuing: Features

• Speed

• Atomic

• Consumers

Queuing: Features

• Speed

• Atomic

• Consumers

• GC

Queuing: Features

•Consumers

Queuing: Features

•Consumers

MongoDB RabbitMQ

Mongo Wire Protocol AMQP

en.wikipedia.org/wiki/State_of_matter

Queuing: Features

•Atomic

en.wikipedia.org/wiki/State_of_matter

Queuing: Features

•Atomic

MongoDB RabbitMQ

findAndModify consume/ack

Queuing: Features

•Speed

Queuing: Features

•GC

Queuing: Features

•GC

MongoDB RabbitMQ

☹ consume/ack

It’s a little different.

Implementation

• Consumers

Implementation

• Consumers

db.runCommand( { findAndModify : <collection>, <options> } )

Implementation

• Consumers

db.runCommand( { findAndModify : <collection>, <options> } )

{ query: { hats: 5 } }

query: filter (WHERE)

Implementation

• Consumers

db.runCommand( { findAndModify : <collection>, <options> } )

{ sort: { added: -1 } }

sort: selects the first one on multi-match

Implementation

• Consumers

db.runCommand( { findAndModify : <collection>, <options> } )

{ update: { $set: {inProg: true, start: new Date()} } }

update: modifier object

Implementation

• Consumers

db.runCommand( { findAndModify : <collection>, <options> } )

remove: true = deletes on returnnew: true = returns modified objectfields: return specific fieldsupsert: true = create object if !exists()

Implementation

• Consumers

Implementation

• Consumers

def run(self): import time, sched sc = sched.scheduler(time.time, time.sleep) self.poll(sc) sc.run()

def poll(self, sc): doStuff() sc.enter(10, 1, self.poll, (sc,))

Implementation

• Consumers

Implementation

• Consumers

from daemon import Daemon class pantalaimon(Daemon): def run(self): # Do stuff

pineMarten = pantalaimon('/path/to/pid.pid') pineMarten.start()

https://github.com/boxedice/python-daemon

Implementation

• Consumers

https://github.com/boxedice/python-daemon

from daemon import Daemon

class pantalaimon(Daemon): def run(self): import time, sched sc = sched.scheduler(time.time, time.sleep) self.poll(sc) sc.run()

def poll(self, sc): doStuff() sc.enter(10, 1, self.poll, (sc,))

Implementation

• GC

Implementation

• GC

now = datetime.datetime.now()difference = datetime.timedelta(seconds=10)timeout = now - difference

queue.find({'inProg' : True, 'start' : {'$lte' : timeout} })

Fault tolerance

Replica sets

Fault tolerance

Replica sets

Basic automatic failover

Fault tolerance

Replica sets

Basic automatic failover

Writing to n slaves

Fault tolerance

Replica sets

Basic automatic failover

Writing to n slaves

Data centre awareness

Scaling

Capped Collections

Scaling

Capped Collections

Extremely fast

Scaling

Capped Collections

Maintain insertion order (FIFO)

Extremely fast

Scaling

Capped Collections

Don’t have to remove()

Maintain insertion order (FIFO)

Extremely fast

Scaling

Sharding

Scaling

Sharding

findAndModify() locks

Scaling

Sharding

Scaling writes across machines

findAndModify() locks

Scaling

Sharding

Scaling

Sharding

Not with capped collections

Problems / Mongo quirks

Problems / Mongo quirks

Meta

Problems / Mongo quirks

Meta

Durability

Problems / Mongo quirks

Meta

Durability

Global lock

Stick with RabbitMQ?

Stick with RabbitMQ?

QoS

Stick with RabbitMQ?

AMQP

QoS

AMQP

Stick with RabbitMQ?

Throttling

QoS

David Mytton

david@boxedice.com

@davidmytton

Woop Japan!

www.mongomonitor.com