8/3/2019 d:evolute Knowledge Message Queues

1/10

Message Queues

* Beanstalkd

* MQ (ZeroMQ)

* RabbitMQ (AMQP)

8/3/2019 d:evolute Knowledge Message Queues

2/10

BeanstalkdBeanstalkd is a Job Queue Server. It listens on a TCP socket, where it receives

jobs from other processes. These jobs stay in beanstalkd's job queue until aworker process reserves, executes them and finally deletes them.

This allows the request processing process to be released, immediately afterthe job was put in the queue.

Web Server beanstalkd Worker(s)

HTTP request

HTTP response

put(do-something) reserve job

delete job

8/3/2019 d:evolute Knowledge Message Queues

3/10

RabbitMQ (AMQP)RabbitMQ is a Message Queue Broker, written in Erlang. It utilizes the openlyspecified AMQP (Advanced Message Queuing Protocol).It features 3 communication schemes: fanout (1-n), topic (1-n, aka pub/sub)

and direct (1-1). Some use cases are: background processing by one or moreworkers, propagating events / updates from & to multiple nodes...

RabbitMQmessage broker

Web Server

Use Case: fanout from web application

HTTP Request

HTTP Responsemessage(published toa fanoutexchange)

Database

XMPP Gateway

...

...

all subscribednodes receivethe message

8/3/2019 d:evolute Knowledge Message Queues

4/10

MQ (ZeroMQ)ZeroMQ is a message queue system, similar to AMQP, though a lot smaller. Itdoesn't rely on a separate broker process. Instead, one of the system's procestakes that role.

Like AMQP, there are multiple communication schemes: request/response (1-1)pub/sub (1-n), ventilator/sink (1-x-1).The ventilator pattern works pretty much the same as beanstalkd (i.e. there areone or more workers, of which the first to shout here gets it), but also has a sinode, where workers push their results.

Client

Server

Request

Response

Publisher

Subscriber

Client

Subscriber Subscriber

Ventilator

Worker Worker Work

request/response publish/subscribe ventilator/sink

Sink

8/3/2019 d:evolute Knowledge Message Queues

5/10

Resources

Beanstalkd:

Homepage: http://kr.github.com/beanstalkd/Ruby Client: http://beanstalk.rubyforge.org/Protocol Specification: https://github.com/kr/beanstalkd/blob/v1.1/doc/protoco

RabbitMQ:

Homepage: http://www.rabbitmq.com/Ruby Clients:- amqp: http://rubygems.org/gems/amqp (EventMachine based)- bunny: http://rubygems.org/gems/bunny

Protocol Information: http://www.amqp.org/MQ:

Homepage: http://www.zeromq.org/Ruby Client: http://www.zeromq.org/bindings:rubyManual: http://zguide.zeromq.org/page:all

8/3/2019 d:evolute Knowledge Message Queues

6/10

1

Message Queues

* Beanstalkd

* MQ (ZeroMQ)

* RabbitMQ (AMQP)

8/3/2019 d:evolute Knowledge Message Queues

7/10

2

BeanstalkdBeanstalkd is a Job Queue Server. It listens on a TCP socket, where it receives

jobs from other processes. These jobs stay in beanstalkd's job queue until a

worker process reserves, executes them and finally deletes them.This allows the request processing process to be released, immediately afterthe job was put in the queue.

Web Server beanstalkd Worker(s)

HTTP request

HTTP response

put(do-something) reserve job

delete job

8/3/2019 d:evolute Knowledge Message Queues

8/10

3

RabbitMQ (AMQP)RabbitMQ is a Message Queue Broker, written in Erlang. It utilizes the openlyspecified AMQP (Advanced Message Queuing Protocol).

It features 3 communication schemes: fanout (1-n), topic (1-n, aka pub/sub)and direct (1-1). Some use cases are: background processing by one or moreworkers, propagating events / updates from & to multiple nodes...

RabbitMQmessage broker

Web Server

Use Case: fanout from web application

HTTP Request

HTTP Response

message

(published toa fanoutexchange)

Database

XMPP Gateway

...

...

all subscribednodes receivethe message

8/3/2019 d:evolute Knowledge Message Queues

9/10

4

MQ (ZeroMQ)ZeroMQ is a message queue system, similar to AMQP, though a lot smaller. Itdoesn't rely on a separate broker process. Instead, one of the system's processes

takes that role.Like AMQP, there are multiple communication schemes: request/response (1-1),pub/sub (1-n), ventilator/sink (1-x-1).The ventilator pattern works pretty much the same as beanstalkd (i.e. there areone or more workers, of which the first to shout here gets it), but also has a sinknode, where workers push their results.

Client

Server

Request

Response

Publisher

Subscriber

Client

Subscriber Subscriber

Ventilator

Worker Worker Worker

request/response publish/subscribe ventilator/sink

Sink

8/3/2019 d:evolute Knowledge Message Queues

10/10

5

Resources

Beanstalkd:

Homepage: http://kr.github.com/beanstalkd/Ruby Client: http://beanstalk.rubyforge.org/Protocol Specification: https://github.com/kr/beanstalkd/blob/v1.1/doc/protocol.txt

RabbitMQ:

Homepage: http://www.rabbitmq.com/Ruby Clients:- amqp: http://rubygems.org/gems/amqp (EventMachine based)- bunny: http://rubygems.org/gems/bunny

Protocol Information: http://www.amqp.org/

MQ:Homepage: http://www.zeromq.org/Ruby Client: http://www.zeromq.org/bindings:ruby

Manual: http://zguide.zeromq.org/page:all