Integrating PostgreSql with RabbitMQ

Integrating PostgreSQL with RabbitMQGavin M. Roy

April 4th, 2014

About MeVP of Architecture

AWeber Communications !

Blog: http://gavinroy.com Twitter: @Crad

GitHub: https://github.com/gmr !

RabbitMQ in Depth (MEAP) http://manning.com/roy

RabbitMQ Primer

About RabbitMQ• Message Oriented Middleware

• Primary identity is defined by AMQP 0-9-1

• Supports other protocols with Plugins: AMQP 1.0, HTTP, MQTT, STOMP, XMPP, UDP, WebStomp, and more

• Written in Erlang/OTP, is Open Source (MPL), and is developed/maintained by Pivotal

Why RabbitMQ?• Create loosely coupled architectures

• Decouple database write operations

• Communicate across application platforms

• Tap into pre-existing message flow for new purposes

• Scale-out clustering for growth, throughput & HA

• Federation for WAN latencies & network partitions

Native AMQP ClientsC

Clojure

Cobol

Common Lisp

Delphi

Erlang

Go

Groovy

Haskell

Java

JavaScript

.NET

OCaml

Perl

PHP

Python

Ruby

Scala

Who Uses It?Agora Games

Chef

Google AdMob

Instagram

MeetMe

Mercado Libre

Mozilla

Nasa

New York Times

NSF

Openstack

Reddit

RabbitMQ Concepts

AMQP Message

Body

Properties

Message Propertiesapp-id

content-encoding

content-type

correlation-id

delivery-mode

expiration

headers

message-id

priority

reply-to

timestamp

type

user-id

Body

Properties

Publishers and Consumers

CP

Multiple Publishers

P

C

P

P

Multiple Consumers

P

C

C

C

AMQ Model

Exchange defines routing behavior

Queuestores messages in memory and optionally on disk

Bindingdefines relationship between exchanges and queues

X

Queue

Binding

Routing Keys• Provided when publishing a message

• Compared against binding keys by exchanges

• Example uses of a Routing Key:

• Connotate the type of message

• Designate the destination of a message

• Categorize the content in the message

Built-In Exchange TypesDirectString matching on Routing Key

Fanout No routing key, messages sent to all bound queues

TopicPattern matching in Routing Key

HeadersNo routing key, string matching in the message headers property

Topic Exchange Binding Keysnamespace.delimited.keys

# Receive all messages

namespace.# Receive all messages in the namespace

namespace.delimited.* Receive all namespace.delimited messages

namespace.*.keys Receive all namespace messages ending in keys

Example Exchange PluginsConsistent HashingDistribute messages via routing key hashed value

Pulse Publish internal RabbitMQ metrics via AMQP

Random Distribute messages across all bound queues randomly

Recent History Stores last 20 messages to any bound queue

Riak Storage Store received messages in Riak"

Script Exchange Calls out to external scripts for routing behavior

Exchange to Exchange Bindings

X

X

X

Queue

Queue

Queue

to RabbitMQ

pg_amqp• PostgreSQL Extension

• User-defined functions to publish via AMQP

• Developed by OmnTI

• Invoke from user-defined function

• Available on PGXN

Using pg_amqp• Has table of AMQP broker connections

• Publishes body only AMQP messages

• Transactional publishing via amqp.publish

• Non-transactional via amqp.autonomous_publish

• Needs some love for additional features

https://github.com/gmr/On-Rabbits-and-ElephantsExample:

SELECT amqp.publish(broker_id, 'name', 'routing-key', 'message');

Publishing a Message

PgSQL Listen Exchange• Exchange that issues LISTEN and publishes received

notifications to bound queues

• When routing messages, bindings are checked for the routing key to match the NOTIFY channel

• Unobtrusive to PostgreSQL environment, native constructs for publishing messages

• Requires PostgreSQL 9.0 and greater

• Body only messages*

Sending a Notificationpsql (9.3.2) Type "help" for help. !postgres=# \timing Timing is on. postgres=# NOTIFY channel_name, ‘Test notification'; NOTIFY Time: 0.160 ms

Receiving a Notificationpsql (9.3.2) Type "help" for help. !postgres=# LISTEN channel_name; !(after NOTIFY issued) !Asynchronous notification "channel_name" with payload "Test notification" received from server process with PID 16749.

Example Notification Message

to PostgreSQL

No direct integrations yet

Chapter 14Creating a pgsql-storage

RabbitMQ plugin

Other ways?

via Consumer apps in the language of your choice

via Apache Flume with RabbitMQ Source, PostgreSQL Sink

Questions?rabbitmq-pgsql-listen-exchange https://github.com/aweber/rabbitmq-pgsql-listen-exchange

pg_ampq http://pgxn.org/dist/pg_amqp/

RabbitMQ in Depth 44% off code pgcf (includes other books as well)http://manning.com/roy/

Follow me on Twitter @Crad