Internet of Things and Multi-model Data Infrastructure

Internet of Things and Multi-model Data Infrastructure

Matthew Aslett, research director

451 Research is an information technology research & advisory companyFounded in 2000

210+ employees, including over 100 analysts

1,000+ clients: Technology & Service providers, corporate advisory, finance, professional services, and IT decision makers

12,500+ senior IT professionals in our research community

Over 52 million data points each quarter

4,500+ reports published each year covering 2,000+ innovative technology & service providers

Headquartered in New York City with offices in London, Boston, San Francisco, and Washington D.C.

451 Research and its sister company Uptime Institute comprise the two divisions of The 451 Group

Research & Data

Advisory Services

Events

© 2015 451 Research. All rights reserved

3


Addressable marketWhile we believe IoT will affect almost every vertical sector, we believe some deserve a special mention:• Industrial automation

• The roots of IoT: increase the efficiency and robustness of manufacturing lines, improve performance and reduce downtime

• Utilities • More efficient energy usage (and billing), predictive maintenance

• Retail • JIT stock-keeping, contextually relevant offers, mobilized PoS terminals

4

Addressable marketWhile we believe IoT will affect almost every vertical sector, we believe some deserve a special mention:• Healthcare

• Health monitoring/alerting for improved treatment, better integrated/connected health equipment

• Transportation and logistics• Improved efficiency, JIT manufacturing and delivery, improved customer service

• Automotive• Improved safety, predictive maintenance, improved fault diagnostics, more

accurate driving data (WARNING: insurance premiums may go up, as well as down)

5

What is the Internet of Your Things?• Companies should identify their own IoT opportunity – if there is one. Specifically,

they should be asking themselves the following questions:

• Are there ‘things’ inside the organization itself that would benefit from greater connectivity?• Is the best use being made of the ‘things’ that are already network-ready and the data that

they create? • Are there ‘things’ outside the organization that would benefit from greater connectivity?• Is there a way to reap some value from customers’, partners’ or suppliers’ smart devices that

would be mutually beneficial?

• Even if the answer to any of the previous questions is ‘yes,’ there are several other considerations to take into account

6


What is the Data of Things?

7

• Metrics and measures. The data that comes from the ‘things’ themselves – measures from sensors such as temperature, humidity, acceleration, vibration, speed, video feeds and so on.

• Transactions. They could include an interaction between two machines, or between a system and a human being. They could include an adjustment to the parameters of a machine or system, such as an alteration to a generator or air conditioning unit.

• Diagnostics. This is the type of data that gives an insight into the overall health of a machine, system or process. Diagnostic data might not only show the overall health of a system, but also serve as an alert that a system is no longer functioning within normal parameters and might need further analysis to determine the root cause.


What is the Data Management of Things?

8

• 451 Research Technology & Business Insight research report

• Written by Jason Stamper (with Matt Aslett)

• Published September 2015

• “The IoT will place unprecedented demands on data storage, processing and analytics. Older systems that were built with fewer, longer-running transactions in mind are already starting to struggle.”



9


Frequency of interaction

Volume of data (per

interaction)

TraditionalEnterprise

applications

IoT

• Traditionally, most transactional systems were designed to be able to cope with one or two transactions every few minutes – at the most

• A sensor or smart device could potentially generate data that needs to be handled by backend systems in some way every millisecond.

• Each individual reading from a sensor might translate into a relatively small amount of data


10


Frequency of interaction

Volume of data (in total)

TraditionalEnterprise

applications

IoT

• Each individual reading from a sensor might translate into a relatively small amount of data, but there are hundreds or thousands being generated each second.

What is the Data Management of Things landscape?

11


• A variety of productsand technologies havea part to play to capturing,processing, storing andanalyzing data from the IoT, fromthe edge, to the datacenter, to thebusiness analyst’s desktop.

• A challenge for processing data from the IoT is being able to ingest data from multiple sourcesin multiple different formats.

Multi-model and multi-mode

12


The database market has been dominated for 40 years by the relational database model (and SQL)

Operational Relational(SQL)


13


The database market has been dominated for 40 years by the relational database model (and SQL) – typically with separate databases for operational and analytics workloads


Analytic Relational(SQL)


14


Emerging databases take advantage of in-memory and advanced processing performance to deliver combined operational and analytic processing



Multi-modeCombined operational and

analytic processing


15


Polyglot persistence drove the expansion of the database market with NoSQL - specialists databases for specialist purposes and multiple data models




analytic processing

Key/value store

Graph database

Document store(JSON)

Wide-column store


16


The use of multiple databases to support an individual application can lead to operational complexity and inflexibility driven by interdependence




analytic processing

Key/value store

Graph database


Wide-column store




analytic processing

Key/value store

Graph database


Wide-column store


17


Multi-model enables the flexibility of polyglot persistence without the operational complexity by supporting multiple data models

Multi-model databasesSupport a combination of the

various individual NoSQL data models.



Key/value store

Graph database


Wide-column store

Others:geospatial,search


18


Multi-model enables the flexibility of polyglot persistence without the operational complexity by supporting multiple data models

Multi-model, multi-mode databasesSupport a combination of the SQL and NoSQL data models

(especially JSON, key/value) as well as other specialist workloads.

Conclusions/recommendations• Companies will need to invest in new data processing and analytics technologies if they

are to keep up with the scale, speed and selection of data types they need to analyze.

• In the area of data platforms and analytics, there is no one-size-fits-all, and no silver bullet. It’s likely that many IoT projects will require a number of data platform and analytic technologies, while in many cases still needing some level of interoperability with existing investments.

• Data platform technologies are coalescing, however, and multi-model and multi-mode products are evolving to fulfill a greater spectrum of the requirements.

• Start to look for the Internet of Your Things.

• Ensure that there is a sound business case for any new investment.

19


Implementing a Multi-model and Multi-mode Approach

In-Memory Distributed Relational

Performance for high concurrency, across models

Scale across compute, and storage resources

Full transactional SQLinserts, updates, deletes

Native analytics support

Built-in enterprise ecosystem integration

451 on the Adoption of In-Memory

Increasingly, companies are moving away from analyzing a set of static data that is days, weeks or months old and closer to trying to take the pulse of databases or even streams of data close to real time.

Source 451: 2015 Trends in Data Platforms and Analytics

Starting with a SQL Engine

Using memory and distributed systems to scale SQL

Retain SQL mathematics and query language for immediate analytics

Small amount of up-front attention (schema on write)

JSON datatype with online ALTERTABLE

Understanding SQL Implementations

SQL as a Layer Full Transactional SQL

Query layer Database

Implementations on top of Hadoop/HDFS SQL Engines

Analytics only No inserts, updates, deletes

Transactions and Analytics Full inserts, updates, deletes

Destination is a report Destination is an operational application to run your business

Expanding Multi-Model Coverage

JSON/Document Then online alter table to

convert JSON to columns

Key-Value Two-column table

Geospatial Points, lines, polygons Concise geo-aware queries

Spark Real-time transformations Streaming and other libraries

451 Take on In-Memory Analytics More employing within organizations look to access

improved BI

Larger market lies in targeting non-developer/ statisticians/data scientists

Putting rapid analytics in the hands of business users not only empowers them

• It could also help to free up technical resources that could be put to better use

Source 451: 2015 Trends in Data Platforms and Analytics

Broad Use Cases

Internet of things

Real-time data pipelines

Predictive Applications

MemSQL Streamliner Real-Time Data Pipeline

Real-Time Application

Apache Spark

STREAMLINER

Extract Transform Load

STREAMLINER

Real-Time Inputs

Fantasy Sport Example

Text input in Ops UI

Read from Kafka topic

BaseballFlat file (full

season), split into chunks (games)

JSON

JSON

Split chunks (games) into

individual events

Two columns: JSON, timestamp

Two columns: JSON, timestamp

Schema specified in transform

Extract Transform Load

Winner

Bul

it-in

Cus

tom

Fast Path to Big Rewards

Availability of fresh real-time data Fast queries

High cost of not capturing data

Commodity Hardware, No SAN, Cloud Deployments Cost mitigation

Thank for Joining UsFollow Matt at https://twitter.com/maslettFollow MemSQL at https://twitter.com/memsqlDownload MemSQL at memsql.com/community