IoT with Apache ActiveMQ, Camel & Spark

IoT with ApacheActiveMQ, Camel and Spark

@burrsutterburr@redhat.com

http://developers.redhat.comhttp://bit.ly/burrIoT1

@burrsutter developers.redhat.com

@burrsutter | burr@redhat.com | http://developers.redhat.com

Agenda

● Business & IT Architecture● IoT Architecture● IETF IoT Use Case● Ingestion: Apache ActiveMQ, Apache Camel● Analytics: Apache Spark● Demos

Business & IT: 80’s to FutureChUI

80’s

Business & IT: 80’s to FutureChUI GUI

80’s 90’s

Business & IT: 80’s to FutureChUI GUI Web

80’s 90’s 00’s

Business & IT: 80’s to FutureChUI GUI Web Mobile

80’s 90’s 00’s 10’s

Business & IT: 80’s to FutureChUI GUI Web Mobile Things

80’s 90’s 00’s 10’s

Ingestion

Sensors

Gateway

Ingestion Analytics

Sensors

Gateway

Dashboards

Workflow

Integration

Ingestion Analytics

Sensors

Actuators

Gateway

Dashboards

Workflow

Integration

Control Messages

Ingestion Analytics

Sensors

Actuators

Gateway

Dashboards

Workflow

Integration

Control MessagesConfiguration

Ingestion Analytics

Sensors

Actuators

Gateway

Dashboards

Workflow

Integration

Ingestion Analytics

Sensors

Actuators

Gateway

Dashboards

Workflow

Integration

Ingestion Analytics

Sensors

Actuators

Gateway

Dashboards

Workflow

Integration

Bananas for Munich - https://tools.ietf.org/html/draft-seitz-ace-usecases-02#section-2.1.1

“The fruit vendor's quality management wants to assure the quality of their products and thus equips the banana boxes with sensors. The state of the goods is monitored consistently during shipment and ripening and abnormal sensor values are recorded. Additionally, the sensor values are used to control the climate within the cargo containers. Since a wrong sensor value leads to a wrong temperature and thus to spoiled goods, the integrity of the sensor data must be assured.”

My embellishment of the key business concerns:Are my bananas over-ripe by the time they arrive at the store?If so, at what point in the 6000 mile journey did they go out of specification?

Monitoring for :● Temperature● Humidity● Accelerometer● GPS

Other Requirements:● Container/vehicle climate control system● Battery operated● Inexpensive (one per crate/box)● Ruggedized enough● Bananas are mostly water - RF interference● Truck->Ship->Train->Truck● On Land Real-Time Monitoring (Cell-based)

Microcontrollers Embedded Linux

Particle.io - Photon WiFi ($19)

LightBlue Bean BLE ($29) Electric Imp WiFi ($29)

TI SensorTag BLE ($29)MetaWear BLE ($45)

Tessel WiFi ($39)

runs JavaScript

Arduino Uno ($25)

Arduino Pro Mini ($10)

Intel Edison ($50)includes WiFi and Bluetooth

Raspberry Pi 2 ($35)+ WiFi ($10)+ Bluetooth ($10)

Onion Omega ($25)includes WiFi

C.H.I.P ($9)includes WiFi, BluetoothArduino Yun ($75)

+ WiFi ($10)

BeagleBone Black ($55)+ WiFI ($10) + BLE ($10) MinnowBoard Max ($145)

ESP8266 ($4)WiFi

IngestionApache ActiveMQ & Apache Camel

Apache ActiveMQ● Top Level Apache Software Foundation project● Client support for Java, C++, C#, Ruby, Python, Perl, JavaScript● Protocols: OpenWire, Stomp, AMQP, MQTT● Embeddable into your Spring and/or Camel apps● Point-to-Point and Pub/Sub Messaging

OpenWireAMQPMQTT

STOMPWebsockets

Point to Point (Queue)

Subscriber/Receiver

Only one receiver of a specific message

Publisher/Sender

Pub Sub (Topic)

Publisher/Sender Topic

Subscriber/Receiver

All subscribers receive the message

Apache CamelApache Camel is an open source Java framework that focuses on making integration easier and more accessible to developers.

● concrete implementations of Enterprise Integration Patterns (EIPs)● connectivity to a great variety of transports and APIs● easy to use Domain Specific Languages (DSLs) to wire EIPs and

transports together● Embeddable in your JVM-based application

Components: MQTT, JMS, File, HTTP, AMQP, SMTP, POP3, HL7, HipChat http://camel.apache.org/components.html

Enterprise Integration PatternsRouter

Translator

Splitter

Aggregator

http://camel.apache.org/enterprise-integration-patterns.html

Filter

Enricher

Wire Tap

Normalizer

Camel Demo

ApacheActiveMQ

(Fuse A-MQ)

ApacheCamel

TI SensorTag - Bluetooth 4.0

MQTT over WiFi

Node.js polls SensorTag over Bluetooth, publishes MQTT over WiFi

Node.js subscriber, if HOTwrites Bluetooth scratch data

MQTT over WiFi

LightBlue Bean - C code

MQTT over WebSocketsto browser

AnalyticsApache Spark

Apache Spark

● Fast: In-memory, runs up to 100x faster than Hadoop Map-Reduce

● Write applications in Java, Scala, Python, R● Batch & Spark Streaming● Spark SQL - SQL & HQL (Hive)● MLlib - machine learning● GraphX

Apache Spark (continued)

● HDFS, S3, Cassandra, Infinispan…● Cluster: Standalone, YARN, Mesos● Simple word count example:

# start the Spark shell (scala REPL)$SPARK_HOME/bin/spark-shell

val f = sc.textFile("data/dracula.txt")val blood = f.filter(line => line.contains("blood"))blood.count()blood.first()

Apache Spark in the Wild

Yahoo● News Personalization● Spark ML: 120 lines of Scala● Replaced 15,000 lines of C++

http://www.datanami.com/2014/03/06/apache_spark_3_real-world_use_cases/

Spark runtime architecture

SparkContext

Driver Program (JVM)

Cluster Manager(standalone, Mesos, Yarn)

Worker Node (JVM)

Task Task

Executor

Worker Node (JVM)

Task Task

Executor

Spark Streaming Demo

ActiveMQ

Spark receives raw events and calculatescheck-in, check-out and retransmit events

Node.js

AMQPAMQP

Stompover

Websockets

BLE Beacons

Gimbal

Estimote

Raspberry Pi 2 +WiFi dongle + Bluetooth 4 dongle

raw check-in/out

Red Hat IoT Project: https://github.com/rhiot/rhiot

IoT with ApacheActiveMQ, Camel and Spark

@burrsutterburr@redhat.com

http://developers.redhat.comhttp://bit.ly/burrIoT1

IoT with Apache ActiveMQ, Camel & Spark

Technology

Messaging With Apache ActiveMQ

Arista Universal Spine - NLNOG · Kudu, Druid, Prometheus, etc Kafka, ActiveMQ, ZeroMQ, RabbitMQ Spark, Storm, Heron, etc Kibana, Grafana Arista Telemetry HBase Kafka CloudVision

A MQ, S M , C CXF - Amazon S3 · WITH APACHE ACTIVEMQ, SERVICEMIX, CAMEL AND CXF Charles Mouillard and ... org.apache.commons.logging.LogFactory; public class. IncidentDAOImpl. implements

Easy Enterprise Integration Patterns with Apache Camel, ActiveMQ and ServiceMix

Introduction to Apache ActiveMQ Webinar Slides

Messaging With ActiveMQ

Introduction to ActiveMQ Apollo

Connecters in ActiveMQ

Apache Camel + Apache ActiveMQ persistence

Easy Enterprise Integration Patterns with Apache Camel ...people.apache.org/~jstrachan/talks/Camel-Dublin07.pdf · Easy Enterprise Integration Patterns with Apache Camel, ActiveMQ

Why real integration developers ride Camels...• Committer on Apache Camel, Apache ActiveMQ, Fabric8 • Recovering consultant • Spent lot of time working with one of the largest

Strategies for Accelerating Identity and Access Management ...€¦ · – A full installation of Apache ActiveMQ – The Apache CAMEL integration package (implementing all the integration

A Pentesters guide to Hacking ActiveMQ Based …...8 A Pentesters Guide to Hacking ActiveMQ-Based JMS Applications WHITE PAPER Apache ActiveMQ Basics ActiveMQ is an open-source, JMS-compliant

Using Apache ActiveMQ with JavaScript

MEAP Edition · I. An Introduction to Messaging and ActiveMQ .....1 1. Introduction to Apache ActiveMQ .....2

Devoxx 2011 integration-camel-cxf-servicemix-activemq

Advanced messaging with Apache ActiveMQ

ZendCon - Integration and Asynchronous Processing with ActiveMQ and Camel

Easy Enterprise Integration Patterns with Apache Camel, …jstrachan/talks/Camel-Dublin... · 2007-11-13 · Easy Enterprise Integration Patterns with Apache Camel, ActiveMQ and ServiceMix

Apache ActiveMQ and Apache ServiceMix