Consuming RabbitMQ at TTL

Consuming Rabbit MQ

JM LaplanteDeveloper @ Trainline.com

What can you use it for?

Pub-Sub

Publisher 1

Subscriber 1

Publisher X

Publisher 2

...

Subscriber 2

Subscriber Y

...

Event A

Event B

Event C

Event A

Event B

Event B

Remote ProcedureCall

Provider

Requester

ResponseCorId: 123

RequestCorId: 123

RequestCorId: 123

ResponseCorId: 123

AsyncJobProcessing

Worker 1

Requester

Worker N...

Job Request 1

...

Job Request N

Job Request 1 Job Request N

Warnings

•RabbitMQ is happier when queues are empty

•RabbitMQ ≠ Data Store

•Connections are precious

•Shared infrastructure*• Poor design decisions have an impact on other components

What are routing topologies?

Publisher

Consumer

PublishingExchange

ConsumingQueue

Routing Topology (Simplest)

Publisher

Consumer

ConsumingQueue

PublishingExchange

Fanout

Routing Topology (Complex)

Publisher

Consumer

ConsumingQueue

PublishingExchange

Fanout

Exchange A Exchange B Exchange C

Subscription Exchange ?

Remote Location

Local Broker B

Remote Consumer

OtherConsumer

Federated FederatedRouting Key

Routing Topology

Proposed approachfor Pub-Sub

Publisher

Consumer

ConsumingQueue

PublishingExchange

Fanout

Routing Exchange

Subscription Exchange

Routing Key Topic

Routing Key

Fanout

What if things fail?

Designing applications for failure•Connections can drop at all times•Nodes can die at all times

•Protect yourself:• Client library should manage connections• Talk to load balancer, not specific nodes• Publish with a circuit breaker library

Designing applications for failure•RabbitMQ delivers messages at least once

RabbitMQ delivers messages at least

once

RabbitMQ

delivers messag

es

at least ONCE

Designing applications for failure•Message processing should be idempotent

•What will happen if a message is processed twice?•No single answer• Immutability goes a long way• Be mindful of side effects

Designing applications for failure•What happens if message processing fails?• Error queues•Dead-letter exchanges

•How will you know?

•How will you replay messages?• Custom-built application• Shovel plugin

How fast is it?

Performance

•RabbitMQ can be very fast• Tens of thousands of messages per second

•Key considerations•Manage connections well•Do you really need publish confirmations?•Do you need message persistence?•Do you need high availability?•How much security should be put in place?

AMQP Transactions vs ‘Publisher Confirms’

Publisher

SynchronousAck

Publisher

AsynchronousAckSLOW FASTER

(100x)

Message Persistence

•Messages that ‘should’ survive broker restart

• Inconsistent publishing times•Disk writes not always consistent (e.g. cheap disk/AWS)

•Do you really need it?

Message Persistence

Website(Matrix)

Add Item To Shopping Basket

Message Persistence

Mobile API

Take Money from Credit Card

Message Persistence

Payment System

Payment Accepted

What about security?

Authentication

• Username/Password for

client applications

• Virtual Host segregation

Client A

ValidCredential

Client A

InvalidCredential

Default vHost

Super Secure vHost

Authorisation

Publisher

Publishing Exchange A

Publishing Exchange B

Consumer

Queue A Queue B

Encryption

•AMQP• Communication channel is not encrypted

•AMQPS• Communication channel is encrypted using SSL• Optional: Exchange signed certificates• Optional: Verify certificates to establish chain of trust

•How much is enough for your data?

How can I monitor my app?

New Relic

•Plugin developed internally•Queue monitoring (Ready/Total messages)• Consumers (Active/Total)•Memory consumption

Circuit Breaker Dashboard

•SCOM alerts

Any .NET client libraries?

Libraries

•RabbitMQ Client API•EasyNetQ (http://easynetq.com/)

• Wrapper around RabbitMQ Client API• Open Source, Well supported, MIT license• Using IAdvancedBus, not IBus*

Questions?