Delayed operations with queues

Yuriy Gerasimov Frédéric G. Marand

Yuriy Gerasimov ygerasimov

• FFW

• Drupal architect & developer

• Contrib 7 modules: services, draggableviews

• Founder at Backtrac.io

Frédéric G. Marand fgm

• OSInet: performance/architecture consulting

• for internal teams at larger accounts

• Core contributor 4.7 to 8.0.x, MongoDB + XMLRPC maintainer + others

• Already 4 D8 customer projects before 8.0.0

• Customer D8 in production since 07/2015

• Frequently adds queueing to larger Drupal projects

Why use queues ?To have websites which are :

• Faster for visitors

• Snappier for editors

• More scaleable

To process time-consuming jobs :

• Video encoding

• High-resolution gallery uploads and processing

Concrete use cases• Prepare content for non-Drupal front-ends

• Anticipate content generation

• Deferred submits, e.g. comments handling

• Slow operations: node saves, previews, image processing

• External data sources: pull, push

• Multi-step operations: batch

Cooking for front-ends

Anticipated content generation

Comment handling

“Pull” data sources (aggregator)

“Push” data sources

Image processing

Job servers• How to get results

• Rerun failed jobs

• Separate queue for failed jobs

• Monitoring queues, workers

• Supervisor

Some implementations

Queue API methods: QueueQueueInterface

• Q::createItem(mixed $data: void

• Q::claimItem($lease_time = 3600: mixed $item

FALSE | stdClass + [item_id => int, data => mixed, created => timestamp]

$lease_time → Assumptions for runner, currently not used

• Q::deleteItem($item): void -> work done

• Q::releaseItem($item): bool

• Q::numberOfItems(): int → best guess, unreliable

• Q::createQueue() / Q::deleteQueue()

ReliableQueueInterface: ordering, single execution

Queue RunnersCore / Contrib

• Core Cron / Elysia Cron / Queue_Runner

• Drush: queue-list / queue-run

Similar limitations:

• Default on in D6 / D7, default off in D8

• Single threaded, single process across queues

Custom runners

• Provided by queue modules or per-project one-offs

• Preemption, parallel execution...

Queue API limitations• Exception handling

• Priority management

• Tagging

• Delay, burying ...

• No monitoring, peek(), deduplication

Questions ?