Improvements in RabbitMQ

Improvements in RabbitMQ Performance and Future Directions

Alvaro Videla - RabbitMQ

@old_sound

github.com/videlalvaro

About MeCo-authored

RabbitMQ in Actionhttp://bit.ly/rabbitmq

What is RabbitMQ

RabbitMQ

RabbitMQ

RabbitMQ• Multi Protocol Messaging Server

• Multi Protocol Messaging Server• Open Source (MPL)

RabbitMQ

• Multi Protocol Messaging Server• Open Source (MPL)• Polyglot

RabbitMQ

• Multi Protocol Messaging Server• Open Source (MPL)• Polyglot• Written in Erlang/OTP

RabbitMQ

Multi Protocol

http://bit.ly/rmq-protocols

http://www.rabbitmq.com/community-plugins.html

Community Plugins

Polyglot

Polyglot• Java• node.js• Erlang• PHP• Ruby• .Net• Haskell

Polyglot

Even COBOL!!!11

Some users of RabbitMQ

• Instagram

• Indeed.com

• Telefonica• Mercado Libre

• NHS• Mozilla

Some users of RabbitMQ

The New York Times on

This architecture - Fabrik - has dozens of RabbitMQ instances spread across 6 AWS zones in Oregon and Dublin.

Upon launch today, the system autoscaled to ~500,000 users. Connection times remained flat at ~200ms.

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-January/032943.html

http://www.rabbitmq.com/download.html

Unix - Mac - Windows

Messaging with RabbitMQ

A demo with the RabbitMQ Simulator

https://github.com/RabbitMQSimulator/RabbitMQSimulator

http://tryrabbitmq.com

RabbitMQ Simulator

Performance Improvements

The Problem

The Problem

./runjava.sh com.rabbitmq.examples.PerfTest \-e exchange_name -t topic -u queue_name \-f persistent -r 5000 -s 1000 -x1 -y0

The Problem

Publish 5000 1kb msgs/sec without consumers

Be able to sustain publish rate during one hour

The Problem

Big throughput drop around 1 minute into the benchmark

The Problem

As soon as Queue Paging kicked inPerformance Dropped

Culprits

• queue_index flush journal

• messages published into the queue index when paging

• memory leaks related to refc binaries

Flushing the QI Journal

• Sparse Array by Message SeqIds

• Publishes, Delivers and Acks

• Queue Index is written in Append Only Segments

• RabbitMQ keeps an in memory Journal of ops to prevent going to disk all the time

Flushing the QI Journal

queue_index_max_journal_entries = 65536

Flushing the QI Journal

entry_to_segment/3 called 65536 times

Flushing the QI Journal

entry_to_segment/3 called 65536 times

Appending data to a binary and then writing that to disk

Solution

• Keep a Sparse Array as a cache of “entry to segments”

• Flush said array to disk whenever the Journal needs to be flushed

Solution

array:to_list(EntriesToSegment).

Solution

array:to_list(EntriesToSegment).

What happens with blank elements?

Solution

array:new([{default, []}, fixed, {size, ?SEGMENT_ENTRY_COUNT}]).

Messages Publishes into the Queue Index

queue_index_embed_msgs_below = 4096

Messages Publishes into the Queue Index

Messages Publishes into the Queue Index

• Implemented Batch Publishing into the Queue Index

• Batch Handling of Delivers and Acks

• Flush Publishes, Delivers and Acks cache every 20.000 messages or when paging finished.

Queue Paging

Alphas -> Betas -> Deltas

binary memory leak• Related to Paging

• discovered with recon

• forced queue process garbage collection after paging

http://ferd.github.io/recon/index.html

The Problem II

The Problem II

Mirror Queue Synchronisation

“is too slow”

The Problem II

Initial tests showed that it took 60 seconds to sync 1MM messages

The Problem II

Original implementation was synching one message at a time

Solution II

Why not implement batch publishing at the queue level?

Batch Publishing• Implemented for Backing Queue Implementations

• rabbit_variable_queue

• rabbit_priority_queue

• rabbit_mirror_queue_master

• rabbit_mirror_queue_slave

Solution II

Initial tests showed that it took 6 seconds to sync 1MM messages

The Problem III

Queues that support offline consumers for 24 hours or more

Solution III• Lazy-Queue concept

• Disable in-memory message cache

• Only load messages in RAM if consumers are present

• Implement Lazy Queues for all Backing Queue implementations

Lazy Queues

https://github.com/rabbitmq/rabbitmq-server/issues/351

Initial tests show stable throughput comparable to “default” queues with persistent messages

Lazy Queues

https://github.com/rabbitmq/rabbitmq-server/issues/351

10MM of 1KB use 1.5MB of RAM vs

1.2GB of RAM for default queues

Lazy Queues

https://github.com/rabbitmq/rabbitmq-server/issues/351

40MB of RAM when a consumer is active

Other Improvements

• moved to erlang.mk

• moving to Ranch/Cowboy and related libraries

• improved management UI performance

• improved rabbitmqctl performance

Other Improvements

• pluggable hashing algorithm for passwords

• queue master location strategies

Future

• add Raft based queue mirroring

• add Raft based cluster formation

• Re-thing rabbitmqctl to allow for extensions

• Tools to recover data from disk

Thanks

Alvaro Videla - RabbitMQ