© 2016 IBM Corporation

IoT Analytics from Edge to Cloud

© 2016 IBM Corporation2

Agenda

What is Edge Edgent (formerly Quarks) overview Database (Informix) on the edge Database in the Cloud Spark

© 2016 IBM Corporation3

What is the Edge? Systems with constrained compute platform

Due to cost, weight restrictions, space constraints, borrowing resources, ... Limited connectivity to central systems

Limited by expense, bandwidth Periods of being disconnected

Can be mobile or static Access to sensors for system being analyzed

Potential to control system based on analysis Typically systems that exist where data(sensor) is generated

© 2016 IBM Corporation4

Edge Examples Vehicle

Car, truck, race car, bike, train, bus, boat, drone, plane, ... Analyze engine sensors to predict/reduce chance of failure Mobile, may lose connectivity

Building/Store/Warehouse HVAC, climate, energy use, motion sensors, ...

Server in machine room Analyze load, cpu temps, rack temps

Raspberry Pi with a couple of sensors Cheap $5+

Smartphone Industrial machine

© 2016 IBM Corporation5

What is the value of analytics at the edge?

• Reduce communication cost• Send relevant data when an event of interest occurs• Heartbeats alone may not contain enough data or be

too late to take action

• React locally to events• More intelligent decision making on the device• Execute analytics while disconnected

• Streaming analysis• Data is constantly being generated• Needing real-time/continuous analyzing

• Collaborate with related devices• Learn from devices with similar characteristics • or location

© 2016 IBM Corporation6

Streaming and In-database Analytics at the Edge

• Streaming analytics• Data is constantly being generated• Needing real-time/continuous analyzing• In-memory analysis of data

• In-database analytics• Store data at the edge efficiently and securely • Provide historical analytics on the data• Visualize the historical data at the edge• Perform complex/advanced analytics on historical

data: windowed aggregates, pattern matching, anomaly detection, etc.

© 2016 IBM Corporation7

Introducing Apache

Apache Edgent (renamed from Quarks)

A community for accelerating Edge Analytics Open Source, incubating at Apache Software Foundation

• http://edgent.incubator.apache.org/

• http://wiki.apache.org/incubator/QuarksProposal

Programming model and runtime for analytics at the edge A programming SDK with Functional Flow API for streaming analytics

• Initial support for Java & Android, Goal is to support multiple languages with priorities driven by the community

A modular, lightweight and embeddable runtime

Initially built by IBM based on 12 years experience with IBM Streams

© 2016 IBM Corporation8

Features Quarks for edge analytics – on device or gateway

Lightweight embedded streaming analytics runtime Analyze events locally on the edge Reduce communication costs by only sending relevant events

Functional Flow API for streaming analytics Per-event and windowed processing with simple analytics and pattern detection

• Map, FlatMap, Filter, Aggregate Connectors

Messaging systems & data stores• MQTT, HTTP, Web Sockets, JDBC, File, Apache Kafka and IBM Watson IoT Platform, Informix

Micro-kernel style runtime with multi-platform support Small-footprint edge devices or sensors

• Including Raspberry Pis or smart phones

© 2016 IBM Corporation9

A Simplified IoT/Industry 4.0 Sensor Data Flow

Sensor Data History

Sensors

In-memory Analytics

Predictive Analytics

Publish / Subscribe

Cloud Infrastructure

HDFS / HadoopStream Processing

Real-time Analytics Operational Analytics Big Data Analytics

(no gateway)

(Gateways) IBM Watson IoT Platform

© 2016 IBM Corporation10

Typical IoT Scenario

Industrial IoT-/Edge-Gateway(s)with specific industry communication stacks

and local Analytic capabilties (in motion / at rest analytics)

Industrial Shop Floor Sensors

Raw Sensor Datavia industry specific automation protocols (e.g. PROFINET, BACnet,OPC UA, Modbus etc.)

Aggregated / Filtered Sensor Datavia standard communication protocols and formats (e.g. MQTT/JSON) intothe Cloud (IBM Bluemix, IBM IoT Platform), IBM PMQ plus on premises solutions etc.

Direct sensor data transfers

© 2016 IBM Corporation11

Informix - The Sensor Data optimized Hybrid Database Superior IoT Sensor Data Performance and Scalability in the Cloud,

on Premises and on Edge Gateways Innovative Sensor Data Storage and Processing w/o the need for costly all-or-

nothing In-Memory database technologies Large Scale and Embedded Sensor Data Benchmarks

Simplified Application Development due to Integration of all Sensor Data relevant data types in one hybrid Database Relational, Time Series, JSON and Geospatial (all joinable!) JSON based Time Series for full Schema Flexibilty Free Developer Editions and Docker Images available

Open, Standardized APIs allow easy Integration into existing Architectures SQL, REST, MongoDB, Node-RED, MQTT, JDBC, ODBC, .NET etc.

Very Cost Effective Operations in all Deployment Models Very efficient HW utilization (less disk space, less CPU & Memory) compared to

other approaches

© 2016 IBM Corporation12

DEMORecorded video of the demo

(https://ibm.box.com/s/hye8csvq14f6cylnk8cqrmtu0h5n2m8h)

© 2016 IBM Corporation13

Live Demo: Sensor Data Flow - Edgent & Informix on a Raspberry Pi

SensortagtoNode-red(Node.js)

Raw sensor data (JSON) captured every 100ms by node-red and sent to Edgent via Websockets

JDBCStream(Insert)

DynamicStreamFilter

Raspberry PiTime Series Data

Data Aggregated and Thresholds checked

Last, dynamic moving averageover 30 seconds

Raw sensor Data

Node-RED Dashboard

WebSockets

© 2016 IBM Corporation14

In-Motion & At-Rest Analytics on the Edge: & Informix DirectProvider tp = new DirectProvider(); Topology t = tp.newTopology("InformixQuarksSensorTagMQTT"); JdbcStreams mydb = new JdbcStreams(t, () -> InformixConnector.getIfxDataSource() , (dataSource) -> dataSource.getConnection() );

MqttConfig mqttConfig = new MqttConfig("tcp://localhost:1883", ""); MqttStreams mqtt = new MqttStreams(t, () -> mqttConfig); TStream<String> msgs = mqtt.subscribe("sensortag/cc2650", 0); msgs = msgs.map(tuple -> new JSONObject(new JSONObject(tuple), new String[] {"tstamp","lux"}).toString()); mydb.executeStatement(msgs, () -> "INSERT into quarks_json_ts_v values (?,?,?)", (g, stmt) -> { String ts = EpochtoTS(new JSONObject(g).getLong("tstamp")); stmt.setString(1, "RAW"); stmt.setString(2, ts); BSONObject bson = new BasicBSONObject("value_r", new JSONObject(g).getDouble("lux")); stmt.setObject(3, new IfxBSONObject(bson)); } );

mqtt.publish(msgs, "values/raw", 0, false);

TStream<String> threshold = mydb.executeStatement(msgs, () -> "select elem.timestamp, elem.value FROM (select GetLastElem(Apply('TSRunningAvg($value_r, 30)',

CURRENT - 30 UNITS SECOND, CURRENT, sensor_values)::TimeSeries(one_value_t)) from quarks_json_ts where sensor_id = ?) AS t(elem);", (g, stmt) -> stmt.setString(1,"RAW"), (g, rs, exc, consumer) -> { if (exc != null) { System.err.println("Unable to execute SELECT: " + exc); } else { rs.next(); Double last = rs.getDouble("value"); String ts = rs.getString("timestamp"); JSONObject jo = null; if (ts != null) { jo = new JSONObject().put("threshold", setThresholdValue(last)).put("tstamp", TStoEpoch(ts)); } consumer.accept(jo.toString()); } } ); mqtt.publish(threshold, "values/threshold", 0, false); msgs = msgs.filter(tuple -> new JSONObject(tuple).getDouble("lux") > thresholdValue);

Insert raw sensor data from stream into an Informix time series

Dynamically calculate moving average from raw time series data

Apply dynamic moving average as a dynamic filter to the stream

Create a new Edgent stream from an MQTT queue

© 2016 IBM Corporation15

Deeper Cloud Analytics

Integrates with centralized analytics systems through IoT scale message hub Any hub Any central system

Informix Hosted Service Push data to Spark for deeper analysis

https://github.com/IBM-IoT/InformixSparkStreaming

© 2016 IBM Corporation16

Heart Monitor Informix

Message Broker

Apache Spark Analytics

Display Results

IOT devices send data into the Informix server

Data Streams from Informix into an MQTT broker

From MQTT Data is streamed into Spark for real-

time Analysis

Results from both Informix and Spark available to the

end user

IoT Spark Streaming Analytics with Informix

© 2016 IBM Corporation17

Sample Code (Edge)

http://github.com/IBM-IoT/edgent-demo

IBM IoT Gateway Kit

http://github.com/IBM-IoT/iot-gateway-kit