ZendCon - Integration and Asynchronous Processing with ActiveMQ and Camel

© 2016 Rogue Wave Software, Inc. All Rights Reserved.

1

Integration and Asynchronous Processing with ActiveMQ and Camel

Justin ReockSr. Director and Lead ArchitectRogue Wave OSS Support


2

Who’s speaking to me?


3

Justin Reock

• Sr. Director and Lead Architect of our OSS Support Team• Way too much experience with ActiveMQ and Camel• Developed our formal instructor-led training• Numerous professional services and training engagements• Early adopter of JMS and Camel• Major FL/OSS evangelist!


4

What are we going to solve?


5


6

Asynchronous processing

• No one likes the spinning wheel of doom hope!• We’re going to talk about offloading work to a backend server• We’re going to do it in a way that’s reliable, flexible, and

guaranteed• Built on industry standards and vetted against thousands of

production systems


7

What else?


8

Quick exercise…

• I’m going to show you three slides• They all have something in common….• See if you can figure out what it is!


9

The Chupacabra…


10

A magical unicorn…


11

Billin

gFu

lfillm

ent

CRME-Com

merce

A homogeneous enterprise landscape…


12

Answer

None of these things exist in real life!


13

I’m listening…


14

Meet: The solutions!

• We’ll be leveraging two related technologies, Apache ActiveMQ and Apache Camel

• Apache ActiveMQ is a JMS implementation, and can help with both federation of systems and asynchronous processing

• Apache Camel is a sophisticated message oriented middleware language, with hundreds of integration components

• But let’s take a step back and answer a couple questions first…


15

What is JMS?


16

Messaging and JMS• Applications often have a need to send information back and forth

to one another in a normalized fashion• Before the advent of messaging systems, it was often difficult if not

impossible to “federate” applications written in disparate languages or residing on heterogeneous platforms

• An early goal of Java was to present a language with the power of C which was not hindered by cross-platform issues

• JMS, the Java Messaging Service, arose out of a growing need to federate very different systems with each other

PHP Application AS400 Application


17

Messaging and JMS• JMS 1.1 – Java Messaging Service

– Intent was to build vendor-neutral specification for sending enterprise messages through Java

– JMS is just a specification– Messaging consists of Messages, Queues, and Topics– Transport endpoints are called Producers and Consumers

• JMS Provider– These are the vendor-specific implementations of JMS– ActiveMQ is a JMS Provider– A JMS Provider must implement the JMS Spec– A JMS Provider forms the framework for facilitating the use of

JMS concepts inside an application– JMS is portable, but JMS providers may not be


18

What is Message Oriented Middleware?


19

Message oriented middleware

• Technically, MOM is any platform that sends and receives messages between distributed applications

• This allows for asynchronous processing, and normalization of data exchanges

• Clients connect to a messaging provider, and send and receive messages via that provider


20

What’s it good for?

• Data normalization makes it a lot easier for disparate applications to trade information

• Just like in web services, heterogeneous systems can agree on a data “contract” and send and receive data in that format

• All the application needs is a client compatible with the messaging provider

• And since messages are allowed to queue up on the message provider, it also makes it very easy to perform asynchronous processing

• Most message providers offer additional benefits, like guaranteed delivery, options for traffic shaping, high availability, and scaling


21

What is ActiveMQ?


22

Apache ActiveMQ

• ActiveMQ is a Java-based multi-purpose messaging middleware solution, called a message broker, designed for high throughput and scale

• It is useful in a number of situations, including aggregating large amounts of data in an efficient and safe manner, federating disparate or heterogeneous applications across an enterprise, and/or moving data between geographically separated data centers or applications

• ActiveMQ supports a number of connection protocols, allowing it to provide connection points between applications coded in different languages, such as Java, C++, .NET, PHP, etc

• Customers commonly migrate to ActiveMQ from commercial middleware such as WebSphereMQ, Oracle AQ, TIBCO EMS, and/or MSMQ


23

Why is it so popular?

• ActiveMQ is a mature and well-adopted platform, with thousands of companies using it, and backed by a strong community

• It is by far the most flexible OSS message broker available, offering integration points for dozens of application frameworks and languages

• ActiveMQ ships with Apache Camel, one of the most revolutionary integration frameworks available, and shares developers and committers with the Apache Camel product

• ActiveMQ can be tuned for massive scale, supporting large data sets, such as those generated by the CERN large hadron collider

• Its configuration can support numerous traffic and implementation models• ActiveMQ has a powerful statistics and metrics engine, exposing crucial

details about the health of the broker and the amount and size of message traffic and client status


24

Where does it fit?

• ActiveMQ acts as integration middleware, sitting between various enterprise applications, data sources, and APIs, and “federating” data between the systems


25

Where does it fit?

• ActiveMQ can also be used as a reliable message buffer• Applications can send messages into ActiveMQ and continue

working, knowing that ActiveMQ will guarantee delivery of the message

• Or, ActiveMQ can be used to broadcast messages to several different systems• ActiveMQ supports transactions between the broker and the client, so that exceptions can be caught and managed

• Using these constructs, you can easily create asynchronous message patterns by sending or receiving messages from ActiveMQ


26

Destination typesQueues• Queues are FIFO buffers of messages • Multiple listeners, or “consumers” can be attached to a single

queue, and can round-robin or otherwise balance processing load• This is called “fanning out”• Client libraries can scale up and down consumers as necessary, but

only one consumer will ever receive a single copy of a messageTopics• These are message broadcast channels• A single message can be broadcast to multiple, single-threaded

“subscribers”• Subscribers can scale using “virtual topics” in ActiveMQ, or “shared

subscriptions” in JMS 2.0


27

Quick demo…

• Let’s integrate a PHP subscriber with a Java publisher• These demos ship with ActiveMQ, and can be found in the

“examples” directory • We’ll be using /examples/openwire/java and

/examples/stomp/php• A Java producer will send 10,000 messages into a topic called

“event”, and a PHP listener will receive those messages


28

Important concepts

• Durability – Describes whether a message will be available to consumers later if they are currently disconnected from the broker

– ActiveMQ defaults all queues to durable, and supports durable topic subscriptions• Persistence – Allows messages to be written to disk storage, so that they will be

available later if the broker should crash– ActiveMQ offers persistence through a choice of high-speed journaling databases,

or RDBMS• High Availability – A factor determining how often a service is available, in this case

how often the broker is up and able to process messages– ActiveMQ includes many native solutions for HA including shared filesystem

options, or true replication through LevelDB/Zookeeper• Horizontal Scaling – Load balancing across multiple brokers and/or message

consumers– ActiveMQ supports highly configurable consumer scaling, along with a Network of

Brokers for chaining multiple brokers together to load balance


29

Working with PHP

• ActiveMQ supports a bevy of transport protocols

OpenWire


30

Working with PHP

• The native protocol for PHP is Stomp, which is a text-based protocol very similar to HTTP

• The Stomp library can be downloaded via pear– pear install pecl.php.net/stomp

• Once the Stomp library has been installed and linked, a Stomp object will be available which can manage every aspect of communicating with ActiveMQ:


31

Additional features

Message groupingScheduled messagesSecurity frameworkMetrics and statisticsNetwork of BrokersInactivity monitoringTransactionsFault toleranceTraffic ShapingPlugin stack

Virtual destinationsTemporary destinationsTransformationLoad balancingClient failoverProperty encryptionDestination policiesDispatch policiesProducer flow controlMessage swapping

And of course… Camel!


32

What is Camel?


33

What’s Camel?

• Camel is an open source MOM language developed by Apache and based on Enterprise Integration Patterns

• Book by Gregor Hohpe and Bobby Woolf• Centers around 60+ common patterns found in Enterprise integration projects

• Defines these common patterns in a vendor neutral way

• Camel aims to provide a language which implements these patterns, in a style modeled after the UNIX pipeline


34

Integration patterns

• Content Based Router– Where a message is routed to disparate endpoints based on message criteria


35

Integration patterns

• Content Enricher– In which a basic message enters the pattern, takes data from

another source, and then adds it to the message


36

Integration patterns• Recipient List

– A message is broadcast to n channels based on message criteria. When this message is broadcast to all channels regardless, it becomes a Multicast pattern


37

Integration patterns• Scatter-Gather

– In which messages are broadcast out to disparate endpoints and results are aggregated into a single message


38

How does it work?

• Camel uses a repeatable, normalized concatenation of “Processor” and ”Message” objects in a group called an Exchange

• There is an ”In” message, a “Processor”, and an “Out” message

Processor

In Out


39

How does it work?

• And, you can chain these Exchanges together – just like piping commands through UNIX – and form a Camel Route

• The “Out” message of a previous Exchange becomes the “In” message of a new Exchange:

P

In Out

P

In Out

P

In Out


40

Camel magic

• Now here’s the magical part – a Processor can be anything you want • Components in Camel work by creating Processors that perform

various business functions• The components can be chained together to build Integration Patterns• For instance, you can have an “FTP” processor that takes the In

message and initiates an FTP session• Or you can have an “SMTP” processor that takes the In message and

sends an email with it• This is in addition to Camel’s many built-in patterns, such as

LoadBalance, Multicast, DataSets, etc.• And there are a lot of components: http://

camel.apache.org/components.html

http://camel.apache.org/components.html

http://camel.apache.org/components.html


41

Embedded scripting languages

• Camel also contains a number of embedded script languages that can be used for various purposes

• Simple: The Simple query language can be used to introspect message headers, payloads, and properties, and is useful when routing messages based on their content

• Xpath: The XPath query language will let you traverse XML structures and query for attribute and entity values, which is great for routing messages based on their XML content

• JSONPath: Just like XPath, except for JSON-structured messages• Groovy: The Groovy scripting language can be used for more

advanced data introspection and manipulation• JavaScript: Yes, you can do entire processors in JavaScript


42

Camel source

• Camel routes can be coded in Java POJO

• Or by using Camel Spring DSL XML files

from("activemq:foo").filter(). xquery("//foo"). to("activemq:bar")

<route xmlns="http://camel.apache.org/schema/spring" id="TimerClient"> <from uri=”activemq:topic:broadcastData"/> <setBody> <constant>This is a test JMS message.</constant> </setBody> <to uri="activemq:Test_ActiveMQ_Route.inQueue"/></route>


43

Camel deployment

• Camel can be deployed in a number of ways:• It can be deployed as a Java Web App using Spring or POJO• Standalone Camel routes can also be deployed using Maven and Camel• Spring Boot can launch Camel routes• Camel can be embedded within another Java application• Or within a Java-based container such as Jetty, Tomcat, JBoss/Wildfly,

and others• ActiveMQ contains the Camel core libraries and can launch and deploy

Camel routes• And OSGI containers like Karaf or its Servicemix counterpart can launch

Camel routes• Standalone Camel routes scale very nicely in microservice containers

as well


44

Camel development

• Major IDEs like Eclipse and IntelliJ can ease Camel development by allowing routes to execute within the IDE

• As of Camel 2.16, comprehensive in-line tools are available which allow auto-completion and even list every parameter of an available component


45

Camel development

• But HawtIO is my current favorite way to prototype with Camel• HawtIO allows for visual debugging of routes, tracing of

message lifecycle, diagram driven development, as well as monitoring


46

Camel monitoring

• Camel has a robust set of JMX instrumentation beans which can be looked at to gain more information on the route

• Any JMX-capable enterprise monitoring solution will be a good fit for Camel


47

Demo: Content based router

• In this example, we will be using a very simple content based router

• In fact, we’ll be using the Simple query language to introspect a JMS header in a message, and routing it to a destination based on the contents of that header

• We will use Camel deployed directly into ActiveMQ, and we’ll write it from scratch against a clean ActiveMQ install


48

Putting it all together


49

The big picture

• So, ActiveMQ provides a platform by which disparate clients can interact with one another in a normalized way

• ActiveMQ’s wealth of clients, including the Stomp, REST, and WebSocket connectors, means almost anything can be a client

• It’s queueing mechanisms make reliable asynchronous messaging a breeze

• Camel can provide sophisticated routing, transformation, and component-driven integration

• And Camel’s huge component library ensures that any application can route through Camel


50

Demo: ActiveMQ, Camel, and PHP

• In this demo, we’ll take a simple web form, and process its data through a synchronous web service as well as an asynchronous queue

• Camel will provide the web service endpoint, and the form will call it synchronously

• PHP’s Stomp client will be used to create a message on a queue for the asynchronous transaction

• Delay has been built in to the messaging flow for both workflows


51

Questions?

Feel free to reach out after, too!

[email protected]

Technology

ZendCon - Integration and Asynchronous Processing with ActiveMQ and Camel