What is ZeroMQ?

Rajan Bhatt1/10/2014

What are we going to cover?

• What is Messaging? • ZeroMQ v/s other Messaging solution• Basics of ZeroMQ• Coding examples • Quiz

What is Enterprise Messaging ?

• Enterprise messaging is a design pattern for loosely coupled system to communicate asynchronously.

• This type of architecture is foundation for “shared-nothing” model.

• Great example of this model in computation is also known as “Actor Model”.

Broad classification of Messaging System

• Broker based Messaging System. – AMQP based system, Implementation RabbitMQ,

ActiveMQ– JMS – Java Message Service.– STOMP – The Simple Text Oriented Messaging

Protocol– MQTT – Light Weight protocol for Sensor networks

• Broker-less Messaging system – ZeroMQ, Nanomsg etc.

AMQP – RabbitMQ

• RabbitMQ is open source message broker software that implements the Advanced Message Queuing protocol. This is written in Erlang language.

• Basic components of AMQP Producer, Exchange, Queue and Consumer

Basic AMQP Model

Publisher

RoutesExchange

Queue

Queue

Consumer n

RoutesPublishes

Consumes Consumer 1

Consumes

In Nutshell - ZeroMQ

• Not MOM ( no broker, no queues, etc..)• “Super” Socket library that acts as concurrency framework• Carries messages across various transport, inproc, IPC, TCP

and multicast.• Connect N-to-N via fan-out, pub-sub, pipeline. Request-reply.• Asynchronous I/O for scalable multicore message-passing

apps.• 40+ languages including C. C++, JAVA, .NET, Python.

In Nutshell - ZeroMQ..

• Is a library that supports certain network communication patterns using socket.

• The “MQ” part comes in because ZeroMQ uses queues internally to buffer messages so that you don’t block your application when sending data.

• When you say, socket.send(…), ZeroMQ actually enqueues a messages to be sent later by a dedicated communication thread.

In nutshell - ZeroMQ

• It does not have overhead of an over-engineered protocol such AMQP

• It can make use of efficient transports such as reliable multicast or Y-suite IPC ytansport

• It makes use of intelligent message batching. This allows ZMQ to efficiently utilize a TCP/IP connection by minimizing not only protocol overhead but system calls.

ZeroMQ – Message Encapsulation

• Encapsulates communication into Messages which may be composed of multiple parts.

• Rather than asking ZeroMQ to receive certain number od f bytes from socket, you ask ZeroMQ to receive a single message.

ZeroMQ - Scalability

• A single ZeroMQ socket can for example connect to multiple end points and automatically load balance messages over them or it can work as some sort of Fan-in, collecting messages from sources through a single socket.

• ZeroMQ follows brokerless design so there is no single point of failure. Combine this with the simplicity and performance and you get something that you can use to make your application distributed.

ZeroMQ Socket – Request- Reply Pattern

• Most basic pattern is client/server model, where a client sends a request and server replies to that request.

• But…• Request type of socket can connect to many

servers and request would be distributes to all connected server.

ZeroMQ Request-Reply Socket Pattern depiction

Zmq client Zmq servers

REQ REP

REP

Request # 1

Request # 2

Reply #1

Reply # 2

Requests are load balanced across server

REP

REP

Demo

• ZMQ Request/Reply Pattern

ZeroMQ – Pub-Sub Pattern

– Publish/Subscribe is classic pattern where senders of messages, called publisher do not program the messages to be sent directly to specific receives, called subscribers. Messages are published without knowledge of what or if any subscribes exists.

Pub-Sub Socket Depiction

ZeroMQ Bus

Pub Socket-1

Sub Socket-3Sub Socket-2Sub Socket-1

Pub Socket-2

Publisher sockets are unaware of number of Subscribers. Subscriber socket can subscribe to multiple of Publisher Socket.

Demo

• ZMQ Publisher/Subscriber Demo

ZeroMQ Socket – Push/Pull Pattern

• Push and Pull sockets let you distribute messages to multiple workers arranged in pipeline. A Push socket will distribute sent messages to its all Pull clients evenly. This is equivalent to producer/consumer model but the results computed by consumers are not sent upstream but downstream to another pull/consumer socket.

ZeroMQ Push-Pull Pattern

Producer Consumers Result Collector

PUSH

PULL

PULL

PULL

PUSH

PUSH

PUSH

PULL

Demo

• ZMQ Push/Pull Socket demo

ZMQ Devices

• ZMQ – sockets are either bind or connect• General pattern is stable part (server) bind

and dynamic part connects to it.

• What do you do when both parts are dynamic ?

• Enter ZMQ device

Types of Devices..

• Queue• Forwarder• Streamer

Client

Client

Client

ZMQ Device

Server

Server

Server

Moving Part Moving PartStable Part

These devises can bind to 2 different ports and forward messages from one end toThe other. The forwarding device can become stable point in your network whereEach component can connect to.

Queue Device

REQ

REQ

REQ

REP

REP

REP

Queue

XREP XREQ

This is the intermediary that sits between clients and servers, forwarding request to server and relaying replies back to client. The ZMQ device takes (ZMQ.QUEUE) and The two sockets bound to well known ports. This is Request/Response broker.

Forwarding Device

PUB

PUB

PUB

SUB

SUB

SUB

Forwarder

XREP XREQ

Just like Queue, Which is like the request-reply broker, For forwarder like the pub-sub Proxy server. It allows both publishers and subscribers to be moving parts and it self becomes stable hub interconnecting them. FORWARDER collects messages from set ofPublishers to and forwards these to a set of subscribers.

Who is using ZeroMQ in production

• Storm platform – communication channel between spouts and bolt.

• Loggly – Cloud based logging platform• Dotcloud – Open source RPC platform called

ZeroRPC based on ZeroMQ messaging.• This is very popular messaging framework for

high frequency financial trading.

Which Messaging is better ? Brokered or Broker-less

• Both messaging patterns have advantages and disadvantages.

• In Enterprise, we see both being used to solve messaging problem which are multidimensional.

• ZeroMQ out-performs other messaging solution in performance, as it is bare-bone framework

Which are other Interesting Open Source Messaging project ?

• Kafka – LinkedIn’s open source message bus, Apache project, used by Tweeter also and many many companies

• NullMQ – Operates over WebSocket, on communication protocol based on STOMP. ZeroMQ semantics in Web Browser.

• NanoMSG-

I know you have questions…

• I may not have answer.. But please ask..• I will find out answers If I don’t know…

Please provide feedback

• Rajan.Bhatt@disney.com.. • Have Fun with ZeroMQ and all messaging..