105

Click here to load reader

Dissecting the rabbit: RabbitMQ Internal Architecture

Embed Size (px)

DESCRIPTION

Talk given at HighLoad++ in Moscow, 2013. Introduces several Erlang patterns and techniques used by RabbitMQ

Citation preview

Page 1: Dissecting the rabbit: RabbitMQ Internal Architecture

Dissecting the Rabbit: RabbitMQ Internal Architecture

Alvaro Videla - RabbitMQ

Page 2: Dissecting the rabbit: RabbitMQ Internal Architecture

Alvaro Videla• Developer Advocate at Pivotal / RabbitMQ!

• Co-Author of RabbitMQ in Action!

• Creator of the RabbitMQ Simulator!

• Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!

• @old_sound | [email protected]/videlalvaro

Page 3: Dissecting the rabbit: RabbitMQ Internal Architecture

About Me

Co-authored!!

RabbitMQ in Action!

http://bit.ly/rabbitmq

Page 4: Dissecting the rabbit: RabbitMQ Internal Architecture

Agenda• Intro to RabbitMQ

• Dive into RabbitMQ Internals

• A day in the life of a message

• RabbitMQ message store

• RabbitMQ behaviours and extensibility

Page 5: Dissecting the rabbit: RabbitMQ Internal Architecture

What is RabbitMQ

Page 6: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ

Page 7: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ

Page 8: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ• Multi Protocol Messaging Server

Page 9: Dissecting the rabbit: RabbitMQ Internal Architecture

• Multi Protocol Messaging Server!• Open Source (MPL)

RabbitMQ

Page 10: Dissecting the rabbit: RabbitMQ Internal Architecture

• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot

RabbitMQ

Page 11: Dissecting the rabbit: RabbitMQ Internal Architecture

• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot!• Written in Erlang/OTP

RabbitMQ

Page 13: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

Page 14: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

Page 15: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

• Java

Page 16: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

• Java!• node.js

Page 17: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

• Java!• node.js!• Erlang

Page 18: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

• Java!• node.js!• Erlang!• PHP

Page 19: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

• Java!• node.js!• Erlang!• PHP!• Ruby

Page 20: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net

Page 21: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net!• Haskell

Page 22: Dissecting the rabbit: RabbitMQ Internal Architecture

Polyglot

Even COBOL!!!11

Page 23: Dissecting the rabbit: RabbitMQ Internal Architecture

Some users of RabbitMQ

Page 24: Dissecting the rabbit: RabbitMQ Internal Architecture

• Instagram

Some users of RabbitMQ

Page 25: Dissecting the rabbit: RabbitMQ Internal Architecture

• Instagram!• Indeed.com

Some users of RabbitMQ

Page 26: Dissecting the rabbit: RabbitMQ Internal Architecture

• Instagram!• Indeed.com!• MailboxApp

Some users of RabbitMQ

Page 27: Dissecting the rabbit: RabbitMQ Internal Architecture

• Instagram!• Indeed.com!• MailboxApp!• Mercado Libre

Some users of RabbitMQ

Page 28: Dissecting the rabbit: RabbitMQ Internal Architecture

• Instagram!• Indeed.com!• MailboxApp!• Mercado Libre!• NHS

Some users of RabbitMQ

Page 29: Dissecting the rabbit: RabbitMQ Internal Architecture

• Instagram!• Indeed.com!• MailboxApp!• Mercado Libre!• NHS!• Mozilla

Some users of RabbitMQ

Page 30: Dissecting the rabbit: RabbitMQ Internal Architecture

http://www.rabbitmq.com/download.html

Unix - Mac - Windows

Page 31: Dissecting the rabbit: RabbitMQ Internal Architecture

Messaging with RabbitMQ

A demo with the RabbitMQ Simulator

https://github.com/RabbitMQSimulator/RabbitMQSimulator

Page 32: Dissecting the rabbit: RabbitMQ Internal Architecture

http://tryrabbitmq.com

Page 33: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Simulator

Page 34: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Internals

Page 35: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

Page 36: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

Page 37: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);!$ch = $conn->channel();!!

$ch->queue_declare($queue, false, true, false, false);!!

$ch->exchange_declare($exchange, 'direct', false, true, false);!!

$ch->queue_bind($queue, $exchange);!!

$msg_body = implode(' ', array_slice($argv, 1));!$msg = new AMQPMessage($msg_body, array('delivery_mode' => 2));!!

$ch->basic_publish($msg, $exchange);

Page 38: Dissecting the rabbit: RabbitMQ Internal Architecture

What happens here?

$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);!$ch = $conn->channel();

Page 39: Dissecting the rabbit: RabbitMQ Internal Architecture

What happens here?

Page 40: Dissecting the rabbit: RabbitMQ Internal Architecture

Erlang

Page 41: Dissecting the rabbit: RabbitMQ Internal Architecture

Erlang App

• Processes (probably thousands)

• They communicate sending messages

• Each process has a message queue (don’t confuse with RabbitMQ queues)

• Virtual Machine has preemptive scheduler

Page 42: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:

http://jlouisramblings.blogspot.ru/2013/01/how-erlang-does-scheduling.html

Page 43: Dissecting the rabbit: RabbitMQ Internal Architecture

Erlang code structure• Modules

• Functions

• Function Arity

• Arguments

M, F, A = Module, Function, Arguments

Page 44: Dissecting the rabbit: RabbitMQ Internal Architecture

rabbit_client_sup.erl-module(rabbit_client_sup).!!

-behaviour(supervisor2).!!

-export([start_link/1, start_link/2, start_link_worker/2]).!!

-export([init/1]).!!

-include("rabbit.hrl").

Page 45: Dissecting the rabbit: RabbitMQ Internal Architecture

rabbit_client_sup.erlstart_link(Callback) ->! supervisor2:start_link(?MODULE, Callback).!!

start_link(SupName, Callback) ->! supervisor2:start_link(SupName, ?MODULE, Callback).!!

start_link_worker(SupName, Callback) ->! supervisor2:start_link(SupName, ?MODULE, {Callback, worker}).!!

init({M,F,A}) ->! {ok, {{simple_one_for_one, 0, 1},! [{client, {M,F,A}, temporary, infinity, supervisor, [M]}]}};!init({{M,F,A}, worker}) ->! {ok, {{simple_one_for_one, 0, 1},! [{client, {M,F,A}, temporary, ?MAX_WAIT, worker, [M]}]}}.

Page 46: Dissecting the rabbit: RabbitMQ Internal Architecture

Supervision Trees

Page 47: Dissecting the rabbit: RabbitMQ Internal Architecture

Supervision tree

• Worker Processes

• Supervisor Processes

• Supervision tree as a hierarchical arrangement of processes

http://www.erlang.org/doc/man/supervisor.html

Page 48: Dissecting the rabbit: RabbitMQ Internal Architecture

rabbit_client_sup

init({M,F,A}) ->! {ok, {{simple_one_for_one, 0, 1},! [{client, {M,F,A}, temporary, !! ! ! ! infinity, supervisor, [M]}]}};

Page 49: Dissecting the rabbit: RabbitMQ Internal Architecture

Child Spec - restart strategies• one_for_one: only restart failing process

• one_for_all: restart failing process and all siblings

• simple_one_for_one: simplified version of one_for_one

• MaxR: maximum restarts allowed

• MaxT: in MaxT seconds

Page 50: Dissecting the rabbit: RabbitMQ Internal Architecture

Child Specificationchild_spec() = {Id,StartFunc,Restart,Shutdown,Type,Modules}! Id = term()! StartFunc = {M,F,A}! M = F = atom()! A = [term()]! Restart = permanent | transient | temporary! Shutdown = brutal_kill | int()>0 | infinity! Type = worker | supervisor! Modules = [Module] | dynamic! Module = atom()

Page 51: Dissecting the rabbit: RabbitMQ Internal Architecture

rabbit_client_sup

init({M,F,A}) ->! {ok, {{simple_one_for_one, 0, 1},! [{client, {M,F,A}, temporary, !! ! ! ! infinity, supervisor, [M]}]}};

Page 52: Dissecting the rabbit: RabbitMQ Internal Architecture

Child Spec - restart

• permanent: should always be restarted

• temporary: should never be restarted

• transient: should only be restarted if terminated abnormally

Page 53: Dissecting the rabbit: RabbitMQ Internal Architecture

Child Spec - shutdown• brutal_kill: child is terminated immediately.

• timeout in seconds: supervisor waits for timeout before terminating children.

• infinity: give enough timeout to children to shutdown its own supervision tree.

Page 54: Dissecting the rabbit: RabbitMQ Internal Architecture

Connection Supervision Tree

Page 55: Dissecting the rabbit: RabbitMQ Internal Architecture

Connection Supervision Tree

Page 56: Dissecting the rabbit: RabbitMQ Internal Architecture

Connection Supervision Tree

Page 57: Dissecting the rabbit: RabbitMQ Internal Architecture

Connection Supervision Tree

Page 58: Dissecting the rabbit: RabbitMQ Internal Architecture

Connection Supervision Tree

Page 59: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

Page 60: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

Page 61: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

Page 62: Dissecting the rabbit: RabbitMQ Internal Architecture

Intermezzo: pattern matchingcheck_user_id_header(#'P_basic'{user_id = undefined}, _) ->! ok;!check_user_id_header(#'P_basic'{user_id = Username},! #ch{user = #user{username = Username}}) ->! ok;!check_user_id_header(#'P_basic'{user_id = Claimed},! #ch{user = #user{username = Actual,! tags = Tags}}) ->! case lists:member(impersonator, Tags) of! true -> ok;! false -> precondition_failed(! "user_id property set to '~s' but authenticated user was "! "'~s'", [Claimed, Actual])! end.

http://videlalvaro.github.io/2013/09/rabbitmq-validating-user-ids-with-erlang-pattern-matching.html

Page 63: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:

http://videlalvaro.github.io/2013/09/rabbitmq-validating-user-ids-with-erlang-pattern-matching.html

Page 64: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

Page 65: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

Page 66: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

We have a list of queues where the channel will deliver the messages

Page 67: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

Page 68: Dissecting the rabbit: RabbitMQ Internal Architecture

A day in the life of a message

If consumer ready

Page 69: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Message Store• Made specifically for messaging

• Keeps messages in memory and (sometimes) on disk

• Bounded by disk size

• per node message store (transient, persistent)

• per queue “queue index”

Page 70: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Message Store

• Message written to disk when:

• Message published as persistent (delivery_mode = 2)

• Memory pressure

Page 71: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:

Page 72: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:Why a custom message store?

!

http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/

Page 73: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:Why a custom message store?

!

http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/

How the message store compacts files !

http://hg.rabbitmq.com/rabbitmq-server/file/56d190fd4ea3/src/rabbit_msg_store.erl#l181

Page 74: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:Why a custom message store?

!

http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/

How the message store compacts files !

http://hg.rabbitmq.com/rabbitmq-server/file/56d190fd4ea3/src/rabbit_msg_store.erl#l181

How the message store reacts to memory pressure !

http://hg.rabbitmq.com/rabbitmq-server/file/56d190fd4ea3/src/rabbit_variable_queue.erl#l35

Page 75: Dissecting the rabbit: RabbitMQ Internal Architecture

Credit Flow

Page 76: Dissecting the rabbit: RabbitMQ Internal Architecture

Prevent processes from overflowing each other’s mailboxes

Page 77: Dissecting the rabbit: RabbitMQ Internal Architecture

Credit Specification

{InitialCredit, MoreCreditAfter}

Page 78: Dissecting the rabbit: RabbitMQ Internal Architecture

Credit Specification

{200, 50}

Page 79: Dissecting the rabbit: RabbitMQ Internal Architecture

Process A sends messages to B

Page 80: Dissecting the rabbit: RabbitMQ Internal Architecture

Process will

• Grant more credit

• Block other processes

• Delay granting credits

Page 81: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:

http://videlalvaro.github.io/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html

Page 82: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours

Page 83: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• Add a common interface for different components

Page 84: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• Add a common interface for different components

• Exchanges

Page 85: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• Add a common interface for different components

• Exchanges

• Queues

Page 86: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• Add a common interface for different components

• Exchanges

• Queues

• Decorators

Page 87: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• Add a common interface for different components

• Exchanges

• Queues

• Decorators

• Authentication methods

Page 88: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• Add a common interface for different components

• Exchanges

• Queues

• Decorators

• Authentication methods

• Add extensibility

Page 89: Dissecting the rabbit: RabbitMQ Internal Architecture

rabbit_exchange_type-module(rabbit_exchange_type).!!-callback description() -> [proplists:property()].!!-callback serialise_events() -> boolean().!!-callback route(rabbit_types:exchange(), rabbit_types:delivery()) ->! rabbit_router:match_result().!!-callback validate(rabbit_types:exchange()) -> 'ok'.!!-callback validate_binding(rabbit_types:exchange(), rabbit_types:binding()) ->! rabbit_types:ok_or_error({'binding_invalid', string(), [any()]}).

Page 90: Dissecting the rabbit: RabbitMQ Internal Architecture

• You can add your own exchange type via plugins

• consistent hash exchange

• random exchange

• recent history exchange

• riak exchange

rabbit_exchange_type

Page 91: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• rabbit_auth_backend (config) • rabbit_msg_store_index (config) • rabbit_backing_queue (config)

Page 92: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• rabbit_auth_backend (config) • rabbit_msg_store_index (config) • rabbit_backing_queue (config)

[{rabbit,! [{auth_backends, [rabbit_auth_backend_http, ! rabbit_auth_backend_internal]}]! }].

Page 93: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• rabbit_auth_mechanism (registry) • rabbit_exchange_decorator (registry) • rabbit_exchange_type (registry) • rabbit_mirror_queue_mode (registry) • rabbit_policy_validator (registry) • rabbit_queue_decorator (registry) • rabbit_runtime_parameter (registry)

Page 94: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Behaviours• rabbit_auth_mechanism (registry) • rabbit_exchange_decorator (registry) • rabbit_exchange_type (registry) • rabbit_mirror_queue_mode (registry) • rabbit_policy_validator (registry) • rabbit_queue_decorator (registry) • rabbit_runtime_parameter (registry)

-rabbit_boot_step({?MODULE,! [{description, "exchange type direct"},! {mfa, {rabbit_registry, register,! [exchange, <<"direct">>, ?MODULE]}},! {requires, rabbit_registry},! {enables, kernel_ready}]}).

Page 95: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Federation Plugin• Message replication across WANs

• Uses an exchange decorator

• Uses a queue decorator

• Uses parameters

• Uses policies

Page 96: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Federation Plugin

rabbitmqctl set_parameter federation-upstream my-upstream \!'{"uri":"amqp://server-name","expires":3600000}'

Page 97: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Federation Plugin

rabbitmqctl set_parameter federation-upstream my-upstream \!'{"uri":"amqp://server-name","expires":3600000}'

rabbitmqctl set_policy federate-me "^amq\." '{"federation-upstream-set":"all"}'

Page 98: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Federation Plugin

-module(rabbit_federation_queue).!!

-rabbit_boot_step({?MODULE,! [{description, "federation queue decorator"},! {mfa, {rabbit_registry, register,! [queue_decorator, <<"federation">>, ?MODULE]}},! {requires, rabbit_registry},! {enables, recovery}]}).!!

-behaviour(rabbit_queue_decorator).

Page 99: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ Federation Plugin

-module(rabbit_federation_exchange).!!

-rabbit_boot_step({?MODULE,! [{description, "federation exchange decorator"},! {mfa, {rabbit_registry, register,! [exchange_decorator, <<"federation">>, ?MODULE]}},! {requires, rabbit_registry},! {enables, recovery}]}).!!

-behaviour(rabbit_exchange_decorator).

Page 100: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:

http://www.rabbitmq.com/federation.html

Page 101: Dissecting the rabbit: RabbitMQ Internal Architecture

Read More Here:Making sure the user provided the right behaviour: !

http://videlalvaro.github.io/2013/09/rabbitmq-internals-validating-erlang-behaviours.html

How RabbitMQ prevents arbitrary code execution: !

http://videlalvaro.github.io/2013/09/rabbitmq-sanitzing-user-input-in-erlang.html

RabbitMQ Boot Step System: !

https://github.com/videlalvaro/rabbit-internals/blob/master/rabbit_boot_process.md

Page 102: Dissecting the rabbit: RabbitMQ Internal Architecture

RabbitMQ uses Erlang’s features to be robust, fault tolerant and very

extensible

Page 103: Dissecting the rabbit: RabbitMQ Internal Architecture

Go grab the source code!

http://hg.rabbitmq.com/rabbitmq-server/

Page 104: Dissecting the rabbit: RabbitMQ Internal Architecture

Questions?

Page 105: Dissecting the rabbit: RabbitMQ Internal Architecture

ThanksAlvaro Videla - @old_sound