Message Queues and Drupal

About Me

Name: Brian Altenhofel

IRC: VeggieMeat

Twitter: @BrianAltenhofel

Drupal: #550616

What I Do: Commerce, Third-party APIs, Migrations, Devops, anything with data...

Favorite Tools: Vim, Scotch, Padron Serie 1926 No. 1 Maduro

Commonly used message queues include Redis RabbitMQ IronMQ (service) Amazon SQS/SNS

What are message queues?

- a datastore for messages to be consumed like a mailbox

- used for parallel processing, ensuring that a task is performed without blocking the user experience

- scalability typically think vertical (server size) or horizontal (more servers) queues allow you to scale time and smooth spikes... great for rate-limited APIs

- decoupling

What are message queues?

ProducerProducerProducerQueueConsumerConsumerConsumer

PublisherPublisherPublisherQueueSubscriberSubscriberSubscriber

Many configurations possible... two most common

Producer/Consumer producers create messages, consumers claim them when they begin working on them, release them on failure, delete on completion i.e. Parallel processing

Publisher/Subscriber publishers create messages, subscribers consume them as long as they are in the queue, message eventually expires and disappears from queue i.e. Data feeds

You already use queues.

- uploading videos that need to be encoded

- GitHub uses queues extensively... most of the work their service does is in the background

- Drupal. Several modules implement queues (i.e. Notifications, Batch API) every time you enable a module, it gets added to a queue to check for updates

In Drupal...

Queues are pluggable

Backends available on Drupal.org:

AWS SQS http://drupal.org/project/aws_sqsBeanstalkd http://drupal.org/project/beanstalkdRedis http://drupal.org/project/redisSTOMP http://drupal.org/project/stomp

(And.... still in sandbox but ready to go...)Openstack Marconi - http://drupal.org/sandbox/BrianAltenhofel/2107851

Using a Different Queue

As easy as:

Global:

$conf['queue_default_class'] = 'MarconiQueue';

Specific queue

$conf['queue_class_{queue_name}'] = 'MarconiQueue';

Queues and Drush

drush queue-list

drush queue-run [queue_name]

Queue-list will list all available queues to run

Queue-run will run them (but will always delete items) advantage over cron is that cron processes can only be run 1 at a time queue-run can be invoked as much as you want

Code.

Code.

function my_module_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}

Code.

function my_module_create_queue_item() { $queue = DrupalQueue::get('my_queue'); $queue_data = array( 'my_key' => 'my_value', );

return $queue->createItem($queue_data);}

Code.

function my_module_count_queue_items() { $queue = DrupalQueue::get('my_queue'); return $queue->numberOfItems();}

Code.

function my_module_process_queue_item() { $queue = DrupalQueue::get('my_queue');

$item = $queue->claimItem(30);

if ($item && $item['my_key'] == 'my_value') { $queue->deleteItem($item); } else { $queue->releaseItem($item); }}

Code.

function my_module_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'my_module_process_queue_item', 'time' => 60, // Time for cron to spend on this queue, not // the lease time. );

return $queues;}

Note: this is only necessary if Drupal is also acting as the consumer/subscriber. This is not necessary is Drupal is just producing or publishing messages to a queue to be consumed by something else.

It's really that simple.

Debugging

watchdog() is a Drupaler's best friend

Log everything (for both your sanity and insanity)

(probably want to ship these logs if you're processing a lot of items)

(if you want to get started on that):
http://www.youtube.com/watch?v=p0Av29yaBEI

Monitoring

Monitor queue depths with your favorite tool (Nagios, Zabbix, Zenoss, etc.)

In a pinch, you can get an idea with $queue->numberOfItems().

Take action if queues get too deep

Actions: Send alert with PagerDuty... spin up new workers...

Any questions?

@BrianAltenhofel

IRC: VeggieMeat (MANY channels)

brian.altenhofel@vmdoh.com