Upload
andrea-giuliano
View
2.985
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Choosing the right way to process data might become a strategic and non trivial decision for many kind of applications. Especially in applications where an high percentage of the time is spent elaborating information behind the scenes. There are different message queuing systems designed to manage and process data asynchronously. Using simple messages, it becomes possible to define many types of queue patterns, from the straightforward solution to the more complex one like routing, publisher/subscriber and topic. The purpose of the talk is to show how to approach the different scenarios in php, adding value to your application.
Citation preview
A S Y N C H R O N O U S D ATA P R O C E S S I N G
A N D R E A G I U L I A N O@bit_shark
# C A K E F E S T
T H E P R O B L E M
P R O D U C T C ATA L O G G E N E R AT O R
• 30k+ records in db representing products
• dispose records in pages (with a certain order)
• generate a 300+ pages PDF catalog
T H E P R O B L E M
H E AV Y O P E R AT I O N S T O D O H E R E
T H E P R O B L E M
U S E R S D O N ’ T W A N T T O W A I TT H E P R O B L E M
U S E R S D O N ’ T W A N T T O W A I TT H E P R O B L E M
A N D I F T H E Y H AV E T O T H E Y D O N ’ T WA N T T O
G E T S T U C K !
T H E P R O B L E M
S Y N C H R O N Y
time
process A
process B
Request
Response
blocked
T H E N E E D S
P R E V E N T T I M E O U TT H E N E E D S
E V E N T U A L LY D E L I V E R YT H E N E E D S
“sooner or later your job will be processed.”
“ E V E R I T H I N G ’ S G O N N A B E A L R I G H T ” N O T I F I C AT I O N
T H E N E E D S
T H E N E E D S
A S Y N C H R O N Y
time
process A
process B
Request
I N T E R O P E R A B I L I T YT H E N E E D S
A D VA N C E D M E S S A G E Q U E U I N G P R O T O C O L
T H E S O L U T I O N
H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce Consumes
B R O K E R
H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
Exchange
Consumes
B R O K E R
H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
ExchangeRoutes
Consumes
B R O K E R
H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
Exchange Queue
Routes
Consumes
B R O K E R
T H R O U G H C O M P O S E R
I N S TA L L A M Q P L I B R A R Y
{ "require": { "videlalvaro/php-‐amqplib": "@stable", ... } }
$ composer.phar install
D I F F E R E N T S C E N A R I O S
P R O D U C E R / C O N S U M E RS C E N A R I O
S C E N A R I O
P R O D U C E R / C O N S U M E R
P R O D U C E R C O N S U M E R
Q U E U E
S C E N A R I O
P R O D U C E R / C O N S U M E R
use PhpAmqpLib\Connection\AMQPConnection; use PhpAmqpLib\Message\AMQPMessage; $connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel();
S E T T I N G U P C O N N E C T I O N
C R E AT E A C H A N N E L
S C E N A R I O
P R O D U C E R / C O N S U M E R
P R O D U C E R
Q U E U E
passive durable exclusive auto-‐delete $channel-‐>queue_declare('catalog', false, false, false, false); foreach ($catalog-‐>getPages() as $page) { $message = new AMQPMessage($page); $channel-‐>basic_publish($message, '', 'catalog'); } $channel-‐>close(); $connection-‐>close();
S C E N A R I O
P R O D U C E R / C O N S U M E R
C O N S U M E R
Q U E U E
$connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); $channel-‐>queue_declare('catalog', false, false, false, false); $callback = function($msg) { $msg-‐>body-‐>generatePdf(); }; $channel-‐>basic_consume('catalog', '', false, true, false, false, $callback); while(count($channel-‐>callbacks)) { $channel-‐>wait(); } $channel-‐>close(); $connection-‐>close();
S C E N A R I O
M U LT I P L E C O N S U M E R S
P R O D U C E R
C O N S U M E RQ U E U E
C O N S U M E R
…
S C E N A R I O
M U LT I P L E C O N S U M E R S
P R O D U C E R
C O N S U M E RQ U E U E
C O N S U M E R
…
parallelize work
easy scalability
C O N S U M E R S C A N D I E
M E S S A G E A K N O W L E D G E M E N T
S A F E T Y
M E S S A G E A C K S
$callback = function($msg){ $msg-‐>body-‐>generatePdf(); $msg-‐>delivery_info['channel']-‐>basic_ack($msg-‐>delivery_info['delivery_tag']); }; !$channel-‐>basic_consume('catalog', '', false, false, false, false, $callback);
S W I T C H A C K O N
D O N ’ T F O R G E T T O S E N D A C K S
T H E B R O K E R C A N D I E
D U R A B I L I T Y
D U R A B I L I T Y
M A R K T H E Q U E U E A N D T H E C H A N N E L
$channel-‐>queue_declare('catalog', false, true, false, false);
In order to achieve durability
$message = new AMQPMessage($data, #the message is now persistent array('delivery_mode' => 2) );
the queue must be declared durable
the message must be marked as persistent
Q O S P O L I T I C S
Q O S P O L I T I C A
B E FA I R
C O N S U M E R
C O N S U M E R
for certain instances of the Round Robin dispatching
B R O K E R
#1#3
#5
#2#4#6
Q O S P O L I T I C S
B E FA I R
$channel-‐>basic_qos(null, 1, null);
C O N S U M E R
C O N S U M E R
the broker sends messages only when it receives acks
B R O K E R
#1
#3
#3 #2#4
P U B L I S H / S U B S C R I B ES C E N A R I O
S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
FA N O U T E X C H A N G E
$connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); $channel-‐>exchange_declare('chatRoom', 'fanout', false, false, false);
Exchange
…
Setting up the connection and declare the fanout exchange
S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Exchange
…
$data = getAMessageToSendInTheRoom(); $msg = new AMQPMessage($data); $channel-‐>basic_publish($msg, 'chatRoom'); $channel-‐>close(); $connection-‐>close();
P R O D U C E R
Produce
Publishing a message to the exchange
N O W I T ’ S T H E S U B S C R I B E R ’ S T U R N
S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Q U E U E B I N D I N GExchange
…
//connection setted up !list($queue_name, ,) = $channel-‐>queue_declare("", false, false, true, false); $channel-‐>queue_bind($queue_name, 'chatRoom');
Bind the Queue on the Exchange
amq.gen-A7d
bind
bind
amq.gen-Sb4
S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Exchange
…
amq.gen-A7d
$channel-‐>basic_consume($queue_name, '', false, true, false, false, 'readMessage'); $channel-‐>close(); $connection-‐>close();
C O N S U M E R
Consumes
amq.gen-Sb4
R O U T I N GS C E N A R I O
S C E N A R I O
R O U T I N G
chatRoom
Exchange type=direct
…
amq.gen-A7d
amq.gen-Sb4
Consumer1
ConsumerN
P R O D U C E R
Produce
routing_keys = friends
routing_keys = friends, colleagues
$channel-‐>exchange_declare('chatRoom', 'direct', false, false, false);
S C E N A R I O
R O U T I N G
chatRoom
Exchange type=direct
P R O D U C E R
Produce
Producing messages//connection setted up !$channel-‐>exchange_declare('chatRoom', 'direct', false, false, false); $data = getAMessageToSendInTheRoom('friends'); $msg = new AMQPMessage($data); $channel-‐>basic_publish($msg, 'chatRoom', 'friends'); $channel-‐>close(); $connection-‐>close();
S C E N A R I O
R O U T I N G
//connection setted up //exchange setted up //$queue_name is a system generated queue name !$rooms = array('friends', 'colleagues'); foreach($rooms as $room) { $channel-‐>queue_bind($queue_name, 'chatRoom', $room); } $channel-‐>basic_consume($queue_name, '', false, true, false, false, 'readMessage'); $channel-‐>close(); $connection-‐>close();
Bind a Consumer on different routing keys and consuming messages
chatRoomamq.gen-Sb4
Consumer
routing_keys = friends, colleagues
T O P I CS C E N A R I O
G O F U R T H E R
T O P I C
$channel-‐>exchange_declare('vehicle', 'topic', false, false, false);
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<type>.<vehicle>.<colour>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
A Y E L L O W S P O R T C A R
T O P I C
A Y E L L O W S P O R T C A R
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<sport>.<car>.<yellow>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
T O P I C
A Y E L L O W S P O R T C A R
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<sport>.<car>.<yellow>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
A R E D R A C E M O T O R B I K E
T O P I C
A R E D R A C E M O T O R B I K E
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<motorbike>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
T O P I C
A R E D R A C E M O T O R B I K E
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<motorbike>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
A R E D R A C E C A R
T O P I C
A R E D R A C E C A R
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<car>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
T O P I C
A R E D R A C E C A R
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<car>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
A B L U E C I T Y VA N
T O P I C
A B L U E C I T Y VA N
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<city>.<van>.<blue>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
D I S C A R D E D
R E M E M B E R
P R O C E S S I N G Y O U R D ATA A S Y N C H R O N O U S LY C A N B E D O N E E A S I LY
E V E N I N P H P
C A N M A K E Y O U R ( D E V )L I F E H A P P I E R
Y O U R A P P L I C AT I O N S T R O N G E R
Andrea Giuliano @bit_shark andreagiuliano.it
joind.in/11608Please rate the talk!
C R E D I T S
• https://www.flickr.com/photos/rayofsun/9401226342
• https://www.flickr.com/photos/thereeljames/11376085194
• https://www.flickr.com/photos/ollily/359817111
• https://www.flickr.com/photos/legofenris/4004170937
• https://www.flickr.com/photos/bsabarnowl/10993445723
• https://www.flickr.com/photos/jpott/2984914512
• https://www.flickr.com/photos/kalexanderson/6231391820
• https://www.flickr.com/photos/a2gemma/1448178195
• https://www.flickr.com/photos/dubpics/5619966355
• https://www.flickr.com/photos/umbertofistarol/5747425870
• https://www.flickr.com/photos/streetcarl/6888965017
• https://www.flickr.com/photos/infomastern/12407730413
• https://www.flickr.com/photos/giuseppemilo/11817936944
• https://www.flickr.com/photos/avardwoolaver/7137096221