View
79
Download
0
Category
Preview:
Citation preview
© 2016 IBM Corporation
IBM Analytics
IBM IoT Architecture and Capabilities at the Edge and Cloud
Shawn Moe – Architect, IBM AnalyticsMay 12, 2016
© 2016 IBM Corporation2
Agenda
IoT Architectural Components
Informix for the Intelligent Gateway
Informix for the Cloud
Performance
IoT Developer Resources
© 2016 IBM Corporation3
Simplified IoT Data Flow
Sensor Data History
Sensors
In-memory Analytics
Predictive Analytics
Publish / Subscribe
Cloud Infrastructure
Real-time Analytics
Real-time Analytics Operational Analytics Big Data Analytics
(no gateway)
(Gateways)
HDFS / Hadoop
Big Data Analytics
MessageSight / MQTT
Gateways for local analytics
InfoSphereStreams
Informix / Cloudant / Watson IOT Platform Service / TimeSeries Service
Informix Warehouse Accelerator / DashDB
PMQ / SPSS / Cognos
Softlayer / Bluemix
Watson / DashDB / BigInsights
Informix / Node-Red
© 2016 IBM Corporation4
• Reduces Cost
• Reduces cloud storage by filtering/aggregating/analyzing data locally
• Reduces cloud CPU requirements by precomputing values
• Reduces Latency
• Intelligent gateways can detect and respond to local events as they happen rather than waiting for
transfer to the cloud
• Performs 80% simple operations locally
• Security
• Gateways allow customers to capture and get value from their sensors without sending data to the
cloud
• Protocol Consolidation
• Cloud does not need to deal with the hundreds of IoT protocols
Over time more and more of the IoT processing will move from the cloud to gateway devices
How Do Smart Gateways Help IoT Solutions?
© 2016 IBM Corporation5
What are the Requirements for a Gateway Database?
The database management system must:
Have a small install footprint, less than 100 MB
Run with low memory requirements – less than 256 MB
Use lossless compression to minimize storage space
Have built-in support for common types of IoT data like time series and spatial/GIS data
Simple application development supporting NoSQL, SQL, and REST
Require absolutely no administration
Be able to network multiple gateways together to create a single distributed logical database
The database must be powerful enough to ingest, process and analyze data in real-time
© 2016 IBM Corporation6
What is a Time Series?
• A logically connected set of records ordered by time
What are the key strengths of Informix TimeSeries?
• Native data type resulting in significantly less space requirements
• Typically about 1/3 the space required by other vendors
• Queries run orders of magnitude faster
• Unique optimized storage means more data fits in memory
• Purpose built streaming data loader for sensor data
• Automatically run analytic and/or aggregate functions on new data
• Integrates structured (SQL) or unstructured (JSON) data to store metadata for each time value
• REST/ODBC/JDBC/JSON interfaces available to work with this
• API contains hundreds of manipulation & analytics functions + APIs to create your own
analytics
Sensor Data is TimeSeries Data
© 2016 IBM Corporation7
Traditional Table storage
Informix TimeSeries storage
Meter_ID Time KWH Voltage ColN
1 1-1-11 12:00 Value 1 Value 2 ……… Value N
2 1-1-11 12:00 Value 1 Value 2 ……… Value N
3 1-1-11 12:00 Value 1 Value 2 ……… Value N
… … … … ……… …
1 1-1-11 12:15 Value 1 Value 2 ……… Value N
2 1-1-11 12:15 Value 1 Value 2 ……… Value N
3 1-1-11 12:15 Value 1 Value 2 ……… Value N
… … … … ……… …
Meter_ID Series
1 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
2 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
3 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
4 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
…
Traditional Sensor data storage vs. Informix TimeSeries storage
© 2016 IBM Corporation8
Many IoT applications have a spatial
component to them
• Vehicles, cell phones, even pets… tracking is
common
In these cases both location and time is
important. Typical uses:
• Show me the vehicles that have passed by
location X in the last hour
• Where has my car been over the last few hours?
Informix allows you to combine Time series
and Spatial data in the same query
IoT Requirements for SpatioTemporal Data
© 2016 IBM Corporation9
Simple to use
• Hands-Free operation – No administration
• Supports popular interfaces such as REST & MongoDB as well as traditional SQL
interfaces (ODBC/JDBC)
• Handles SQL and JSON data in the same database
• Seamless data replication and HA solutions to move or copy data where
needed
Performance
• Unique support for TimeSeries and Spatial data
• Stream data continuously into the database
• Run analytics as data arrives
• Dynamically add and update analytics when needed
• Storage is typically 1/3 the size compared to other vendors
Invisible
Agile
Informix is the only database management system perfectly suited to run in Gateways
IBM Informix: The Ideal Database for Gateways
© 2016 IBM Corporation10
What are the IoT Requirements for the Cloud?
• Requirements - similar to gateways (but for different reasons):
• Potentially 1000’s of servers means zero administration is a must
• Data volume adds up very quickly so low storage overhead is required
• Data flows into the cloud continuously and must be processed in real-time
• Must be able to handle time series, spatial, SQL and NoSQL data natively
• Additional requirements
• Must be able to scale-out
• Must be available as a service
The database must be able to ingest, process and analyze data in real-time
© 2016 IBM Corporation11
Why use Informix in the Cloud?
Simple to use
• Hands-Free operation
• Supports REST and MongoDB APIs as well as ODBC/JDBC
• Stores SQL and JSON collections in the same database
• Seamless data replication and HA solutions to move or copy data between Informix implementations in the IoT network
Highly Available
• Close to zero down time
• Partition your data across servers in the cloud
• Dynamically add/remove additional servers
Performance
• Continuous High Performance Analytics
• Specialized support for Time Series and Spatial data
• Build Hybrid applications to work with all types of stored data
• In-Memory query acceleration speeds analysis and analytics to near real time
Invisible
Agile
Resilient
© 2016 IBM Corporation12
Shards: Scale-out your Database across Servers or Gateways
• Distribute data among servers by range or hash partitioning
• Each shard can have an associated secondary server for high availability
• Run queries across all shards or a subset of the shards
• Only shards that could qualify are searched
• Shards are searched in parallel
• Ignores shards that are offline
Shards in a Cloud
© 2016 IBM Corporation13
Every IoT deployment will need to store time series data and possibly spatial data
Bluemix Cloud Services
Watson IoT Platform Service on Bluemix• Quickly and simply add new sensors
• Interface for continuously loading sensor data
• Adapters for MQTT and MessageSight
• Uses Informix for TimeSeries historian
© 2016 IBM Corporation14
Benchmark: Informix vs SQLite
Tests on Intel Quark Informix SQLite
Data loading – high-speed performance (records per
second)950 / 1050 secs
(DK100 / DK50)1
700 secs
(Average)2
Storage space that is required for 1 day of data 275 MB 1200 MB
Aggregation query (seconds) 2 secs 4-25 secs
Moving average (seconds) 25 secs 2592003 secs
Missing interval search (seconds) 2 secs 14-30 secs
1. The two figures for data loading with Informix reflect a slight difference in performance between the DK100 and DK50. DK100 had more
running components causing a drop in performance vs DK50
2. Data loading with SQLite had significant variations in load performance as the database size increased.
3. The moving average result for SQLite is a projected figure that is based on a partial result after 10 minutes.
© 2016 IBM Corporation15
Metric Competitor Informix
Daily processing time
Maximum number of cores used
11 hours
62
5 hour 50 min
32
Maximum amount of memory used 192GB 192GB
Size of database per month of data 15TB 5TB
# Records processed each day 2.88 Billion 2.88 Billion
Billing determinants creation (1/21 of the total meter population)
51,322 ~2 million reads per second
TimeSeries Meter Data Management Benchmark
- 30 million smart meters sending data every 15 minutes- 2.88 billion records inserted each day- Workload: data Ingestion, data cleanup, and a daily billing cycle
© 2016 IBM Corporation16
Smart Building Example
• Fire & smoke sensors
• HVAC sensors
• Door & window sensors
• Lighting sensors
• Water sensors
• Solar panels sensors
• Elevator sensors
• …Dashboard &
local analytics
On-premises
Smart Gateway &
database
Replication of selected
building data – all or part
Deeper or cross-building
analytics + employee data
Watch this demo video on YouTube! (https://youtu.be/q0D3acmXAjo)
© 2016 IBM Corporation17
IBM IoT Smart Gateway Kit
• Designed to quickly get you up and running on ARM or Intel SoC devices, like
Raspberry Pi
• Works with TI SensorTag and Node-RED
• Simple install and configuration of:• Informix database
• Node.js
• Node-red
• TimeSeries nodes
• Bluetooth node.js application sample
• Visualization nodes and dashboard
Clone the GIT repository, run the install script, turn on your SensorTag and you are collecting TimeSeries data in your Gateway!
© 2016 IBM Corporation18
IBM Informix - best fit for IoT architecture
IoT Smart Gateway
IoT Cloud analytics
Supported on a wide array of platforms, including SoC computers
Best in class embeddability
Native support for sensor data - TimeSeries & Spatial data
Native support for unstructured (JSON) data
Ease of application development - REST access
Support to receive IoT data via MQTT protocol
High availability and dynamic scaling
In-memory query acceleration
Summary
© 2016 IBM Corporation19
IBM Smart Gateway kit - https://ibm.biz/BdXr2W
Code samples - https://ibm.biz/BdX4QV
Github - https://github.com/IBM-IoT/
Free Informix Developer Edition - https://ibm.biz/BdXp2g
Informix on Docker Hub https://registry.hub.docker.com/u/ibmcom/informix-
innovator-c/
Informix Developer Edition for Raspberry Pi (32bit)
https://registry.hub.docker.com/r/ibmcom/informix-rpi/
IoT Developers - Get Started!
Docker Hub
© 2016 IBM Corporation21
SolutionThe embedded solution adds to
Eurotech`s IoT gateway hardware and
Everyware Software Framework (ESF) ,
Informix TimeSeries technology, to
capture data and interact with the
devices, sensors and meters in the field.
Everyware Cloud, deployed in the
Cloud or on premise acts as an
M2M/IoT integration platform between
the distributed devices and the IT world.
Combined and integrated these
elements ensure best TCO and
performance in IoT solutions.
Encapsulating complexity of IoT/M2M solution, ensuring scalability and high performance for customers globally.
ChallengeReducing the cost, effort and risk
challenges in IoT/M2M projects.
Ensuring scalability, performance
and efficiency when developing,
deploying and running distributed
devices, sensors and meters in the
field.
Delivering highly integrated solutions
combining powerful database
technology at the edge / gateway and
the center of the network with state of
the art M2M/IoT software and
hardware building blocks.
Recommended