Devising Your Data Movement Strategy for IoT

1Copyright SolaceConfidential

Devising your Data Movement

Strategy for IoTDave McAllister

September, 2016


Data Movement Between Elements of IoT

Top level system considerations

Architectures for different types of data movement

Which standards to use;where and why


Relevant System Requirements

Scale of Devices

Capabilities ofDevices

Message Exchange Patterns

Messaging QoS

TrafficCharacteristics

SecurityIT Application Architecture & Technologies


ProblemStatement

Firewall

Firewall

LoadBalancer

? ? ? ?




Message Bus


IoT Interlude

http://imgs.xkcd.com/comics/driving.png


Firewall

Connection Tier

LoadBalancer

EDGE

FOG

Typical IoT CloudDeploymentArchitecture

Enterprise BusFirewall

CORE


Firewall

Connection Tier

LoadBalancer

EDGE

FOG

System Considerations• Connections, queues,

subscription• MEPs

• Device initiated pub/sub

• Device initiated request/reply

• Cloud app initiated pub/sub

• Cloud app initiated request/reply

• QoS – MQTT QoS0, QoS1• Single point of contact for

apps

Enterprise BusFirewall

CORE


Enterprise Bus

Connection Tier

System Considerations

Load Balancer

o Connections, queues, subscrptns

o MEPs‐ Device-initiated pub/sub‐ Device-initiated

request/reply‐ Cloud app-initiated pub/sub‐ Cloud app-initiated

request/replyo QoS – MQTT QoS0, QoS1o Single point of contact for

apps


Enterprise Bus

Connection Tier

Load Balancer

System Considerationso Connections, queues,

topicso MEPs

‐ Device-initiated pub/sub‐ Device-initiated

request/reply‐ Cloud app-initiated pub/sub‐ Cloud app-initiated

request/replyo QoS – MQTT QoS0, QoS1o Single point of contact for

apps


Enterprise Bus

Connection Tier

Load Balancer

Challenges with Stateless Connect Layero Device Queues

‐ Where is the queue for each device for QoS1 subscriptions?

‐ How does Connection layer find its queue/state? (among the millions….)

o App to device MEP‐ How does Core layer find

Connection layer server for given MQTT device for QoS0?

‐ Where is the Q for a given device for QoS1?

‐ Need massive subscription scalability updated in real time


MQTT Paho

Proprietary Proprietary/JMS/JCA

HTTP/REST In Runtime

AMQP 1.0 Qpid / JMS

CoAP Several OSS

P R O T O C O L A P IRelevantProtocols& APIs


MQTT versus CoAPMQTT CoAP

Many to many (but Pub-sub) One to one

Event based State based

No discovery (Pub-sub) Supports content negotiation

OASIS / ISO IETF

3 levels QoS 2 levels QoS

17confidential

o Device Constraintso Message Exchange

Patternso Messaging QoSo Traffic Characteristicso Security

Fog

18confidential

Fog

MQTT Paho



AMQP 1.0 Qpid / JMS

CoAP Several OSS

P R O T O C O L A P I


Edgeo # Concurrent

Connectionso “Always on”

Connections?o Burstinesso Congestion /

Prioritizationo Multi-protocol?o QoSo Security


MQTT Paho



AMQP 1.0 Qpid / JMS

CoAP Several OSS


Edge


o Less scale, more concentration

o Need enterprise features

o Enterprise grade, popular APIs

o Support streaming & microservices

Core


Core

MQTT Paho



AMQP 1.0 Qpid / JMS

CoAP Several OSS



IoT is all about the data

Capacity &Availability Limits

Big Data RIVERShock Absorber

NetworkProcessingStorage

OutagesUpgrades

InconsistentAggregate

InputStream


PublicClouds

PublicCloudAp

pApp

App

Big Data River

Big Data

Private Cloud

App App App

App App App

PaaS

IaaS

App App App

PublicCloud

On Premise


Enterprise Bus

Connection Tier

Addressing Scalabilityo Command & Control

sending message to a specific device


Enterprise Bus

Connection Tier

Addressing Scalabilityo Command & Control

sending message to a specific device

o Device-initiated request/reply

o Among millions of deviceso Across 2 middleware tierso Need Integrated

addressing between Enterprise Bus & Connection Tier at massive scale


Enterprise Bus

Connection Tier

Addressing Scalability

Load Balancer


Enterprise Bus

Connection Tier

Addressing Scalabilityo Connection Tier

Addressing‐ MQTT Topics

o Enterprise Bus Addressing‐ Other topics & queues

o Need to scale ‐ In an integrated manner‐ Update in real-time‐ Handling bursts of changes


Singapore Land Transport Authority – ERP2http://www.straitstimes.com/singapore/transport/ncs-mhi-to-build-islandwide-satellite-based-erp-for-556m

Routing Server

Payments

Analytics

0 0 0

• 200K Concurrent Connections

• MQTT Termination

• Request/reply forfirmware upgrades

• Push Notifications

• Persistent Messaging

• Shock Absorber

• High Throughput

• Fanout

• Slow Consumer Handling

MQTT JMS

MHI OBU

Next-Gen Electronic Road Pricingo Connect 1.5M vehicles

(200K at a time) in real-time for pay per use road tolls

o Suggest better routes, traffic management – real time and long term and value-added services/analytics

http://www.straitstimes.com/singapore/transport/ncs-mhi-to-build-islandwide-satellite-based-erp-for-556m




Bus Monitoringo Collect real-time

location, loading information and more thousands of buses

o Optimize bus movement, fleet management, avoid bunching, forecast arrival time and more

REST/MQTT

Regulator’s Dashboard

Commuter’s Apps and

RIAs

Analytics

DWH

ESB (if needed)

CEP

JMS

Legacy REST

Legacy Gateway

REST/MQTT


Overall Architecture Requirementso Scalability: connections,

queues, subscriptionso “single bus” connectivity for

both apps & devices – solve the whole problem!‐ Unified addressing between

cloud apps & devices‐ Single point of application

connectivityo Support necessary QoS

o Support all MEPs(?)‐ Device-initiated pub/sub‐ Device-initiated request/reply‐ Cloud app-initiated pub/sub‐ Cloud app-initiated

request/reply

IoT Connectivity Concerns


At the Edgeo Scalability: connections,

queues, subscriptionso Protocols: MQTT, REST,

CoAP(?)o Security:

‐ Authentication & Encryption‐ Per-device authorization

‐ deviceID in topics‐ /in/<devID>/<system> ACL on

/in/<devID>/>

In the Coreo Single, simple point of

contact to talk to any deviceo APIs & protocols to cloud

apps‐ JMS, AMQP, Node.js, REST‐ App-to-device and app-to-app

o Message filteringo Application horizontal

scaling with non-exclusive queues

o Integration with Hadoop

IoT Connectivity Concerns(cont)


Can’t predict coming evolution of IoT,and changing infrastructure isn’t cheap.

Need flexible data movement platform


Strong Architecture• Decoupling of concerns

• Scales to meetneeds of clients

• Right featuresat each layer

Smart Use of Standards• Right standards• Features & exchange patterns

• Optimized for needs of users

• Security• Supported by key products andopen source projects


Thanks!

Dwmcallister [email protected]

Download the free VMR Community Edition at

http://dev.solace.com

Technology

Devising Your Data Movement Strategy for IoT