Real-Time Sensors Data Webinar | SQLstream | July 2013 Series

If you haven’t dialed into the audio portion, please do so now:

U.S.A +1 (646) 307-1721| 789-157-692

T H A N K YO U F O R J O I N I N G ! T H E W E B I N A R I S A B O U T TO S TA R T

Listen to Your Sensors: a Tale of Managing Large-scale, Intelligent Sensor Networks in Real-time

| 2 Copyright © 2013 | +1 877 571 5775 | [email protected]

¤  Explain real-time Big Data and Operational Intelligence

¤  The principles of streaming data management

¤ Share our thoughts, experiences and use cases

¤ Audience Q&A

PROGRAM MISS ION


¤  Introduction (5 min)

¤ Presentation (35 min)

o  An Introduction to Operational Intelligence

o  Solution Architecture for Real-time Sensor Analytics

o  Streaming Big Data management

o  Industry Use Cases

o  Demonstration (Remote sensor monitoring)

o  The Total Cost of Performance

¤ Q | A (20 min)

JULY 16 AGENDA


¤  July 9 2013 |10:00am PST- recording available upon request Analytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence

¤  July 16 2013 |11:00am PST Listen to your Sensors: A Tale of Managing Large Scale Sensor Networks in Real-time

¤  July 23 2013 |11:00am PST Predict and Avert: Using Log File Data to Prevent Cybersecurity and Fraud Attacks in Real-time

¤  July 30 2013 |10:00am PST No more CPR for your CDRs: Meet Real-time Traffic Utilization, Billing and Fraud Detection

T he Opera t i ona l I n t e l l i gen ce Se r i e s


¤ 25 years of experience in database and software applications

¤ Senior positions with Oracle, Hyperion Solutions, Information Builders and Algebraix Data

¤ Recent guest speaker at 2013 Sensors Expo

Today ’s P re sen te r : G lenn Hou t SQL s t ream VP fo r t he Amer i ca s


THE INTERNET OF EVERYTHING Manag i ng La rge - s ca l e Con ne c t ed Se n s o r Ne two r k s

TECHNOLOGY

¤  OPEX & CAPEX

¤  Experience

¤  Loyalty

BUSINESS CASE

¤  Velocity

¤  Volume

¤  Unstructured

REAL-TIME BIG DATA THE NEXT DIGITAL AGE

REAL-TIME

OPERATIONAL

INTELLIGENCE

50 Billion Connected

Devices and Sensors by

2020

(Cisco IBSG)

¤  Wireless

¤  IPv6 Networks

¤  Smart Energy


ABOUT SQLSTREAM

facts

o  Launched 2009

o  Over 1.5M lines of code

o  Multiple deployments across many industries

capabilities

o  Unstructured and structured data

o  Accelerates and extends Hadoop & RDBMS

o  Not limited to SQL

innovations

o  Only true streaming data management platform

o  Only true standard SQL streaming engine

o  Five patents for stream processing

ü  Real-time Operational Intelligence ü  High-velocity machine data. ü  Streaming Big Data Management Platform

¤  TWEET: during and after the webinar, please

use #RTSensorData for live discussions

¤  DIRECT QUESTIONS: please use the box to the

right of your screen

¤  RECORDINGS: an edited version of the webinar

recording will be emailed upon request

Opera t i ona l I n t e l l i gen ce


The In ter net -of -Th ings Explos ion | SOURCES E v e r y t h i n g i s b e c o m i n g i n s t r u m e n t e d ; v a r i e t y

ENVIRONMENTAL TRANSPORTATION NETWORKS

Environmental Monitoring Location-based services Machine-to-Machine

Smart Grid Cars as Sensors Logistics


OPERAT IONAL INTEL L IGENCE I n t eg ra t i ng Opera t i on s and Ana ly t i c s i n Rea l - t ime

As we move toward a real-time business environment, the capability to process data flows swiftly and flexibly will become increasingly important. SQLstream leads the industry in this kind of capability. ”

Robin Bloor Chief Analyst for Bloor Group

Business Intelligence

Operations

Real-time Operational Intelligence Continuous monitoring and analytics

Improve decision-making

Automate operational processes

Environment Smart Grid

Transportation Telematics

M2M Remote Monitoring

”


T he I n fo r ma t i on Va l ue Cha i n

What is happening?

What might happen?

What just happened?

Make stuff happen!


COLLECTION: Massive numbers of sensors

o  Thousands, tens-of-thousands, hundreds-of-thousands,

millions+ of sources

o  Creation of multiple siloes

SCALE: Continuous “fire-hose”

o  Massive data volumes and velocities

o  Information overload

PROCESSING: Actionable intelligence

o  Data “decision” value can drop exponentially with

passing seconds

o  Data velocity and rate-of-change critical for decisions

o  Lack of real-time processing

B ig Data Explos ion| CHALLENGES A n a l y t i c p r o c e s s i n g a t m a s s i v e s c a l e

BIG DATA

MACHINE DATA

HIGH-VELOCITY, LOW-LATENCY INTELLIGENCE


¤ Capabilities

o  Health monitoring of key vehicle systems

o  Frequent monitoring when “exception” system

events occur

o  Real-time “panic” alerts

¤ Objectives

o  Reduce vehicle “walkaway” events by 50%

o  Improve customer satisfaction

o  Reduce warranty costs

o  Future: Potential “paid for” service

R EAL - T IME INTEGRATED VEH ICLE HEALTH MANAGEMENT

Warning! Battery draining fast. Please check for lights left on.


¤ Real-time vehicle diagnostics & alerting

o  Pre-emptive vehicle service

o  Critical event monitoring and alerting

¤ Asset tracking for fleet management

o  Track vehicles, shipments and transit ETA

¤ Real-time driving log

o  Safety, compliance and alerting

¤ Dynamic road tolling

o  Assess tolls based on weight, miles, roads used

VEH ICLE TE LEMAT ICS & HEALTH MONITOR ING

Log


¤  Insurance Programs

o  Breaks on insurance for good drivers

•  Immediate “journey” reports to smart

phones

o  Real-time analytics and alerting

•  Safety, compliance and real-time

monitoring

¤ Geofencing

o  Alerts to smartphones when drivers

travel outside prescribed boundaries

INSURANCE & SENSOR MONITOR ING

Driving Analysis: ……… ………………. ………….


¤  Real-time health diagnostics, monitoring and alerting

¤  Asset tracking and location-based services

¤  Location- and activity-based advertising

¤  Safety, regulatory and compliance

¤  Customer loyalty solutions

¤  Fleet management solutions

¤  Usage-based applications: insurance, tolling, traffic, etc.

¤  Emergency services and responder alerting

REAL - T IME SENSOR APPL ICAT IONS


Information coming continuously or in irregular intervals

o  Sensor data, log file data, health information (from cars, machinery, hardware, people), etc.

Anything that requires immediate attention

o  Alerts that must be acted on now, not in minutes, hours or days

Any process that struggles with batch window latencies

o  Service level agreements (SLA’s) with narrow time-action windows

o  Batch queries that are continually re-run

o  Data cleansing, transformation and loading (ETL)

Massive scale

o  Information inflows so large that they must be continuously processed

R EAL - T IME APPL ICAT IONS | IND ICATORS


Capability Real-Time System Traditional Database / Hadoop

Analytic Processing

•  Continuous queries •  Incremental calculations

•  Batch queries •  Expensive, batch recalculations

Latency from event to action

•  Milliseconds to seconds •  Tens-of-minutes, hours, days •  Significant latency

Rate of Change •  Monitor data velocity and

complex rates-of-change •  Static snapshot of point-in-time

Scalability & Hardware

Requirements

•  Millions to tens-of-millions of events/second

•  Minimal hardware reqt’s

•  Massive data inflow and continual recalculation limits analytic depth

•  Significant hardware reqt’s

Iterative Analytics

•  Cascading analytics within and across events - pipelining

•  Requires myriad views, temporary tables and custom application code

QUERY-THEN-STORE VS. STORE -THEN-QUERY


Category Operational Intelligence Business Intelligence Vehicle

Diagnostics Is my vehicle about to fail? How many vehicles failed in the last 6 months

in the state of New York?

Sensors ALERT - The airbag just deployed! What is the current location, vehicle and driver information?

How many vehicles had airbags that deployed in the past 2 weeks by vehicle type?

Telematics & Asset Tracking

Where is the vehicle right now? Where has it been and how is it moving on current journey?

What was the average speed of all vehicles on various road segments on prior journeys?

Advertising What is a relevant ad to place based on current location, current activity and historical patterns?

What is a relevant ad to place based on historical patterns?

Security Is an intrusion attempt underway? Where have intrusions happened in the past and from what sources?

Fraud/risk Is the current transaction fraudulent?

How many fraudulent transactions occurred at online electronics retailers during the past 3 months?

R EAL - T IME ANALYT ICS VS. H ISTOR ICAL ANALYT ICS


R EAL - T IME , CONT INUOUS OPERAT IONAL INTEL L IGENCE

Real-time alerts, action and visualizations

Enhance real-time data by joining it with historical information

Persist both detail and aggregate data to historical archives


“ R EPLAY” EVENTS TO DEVELOP PRED ICT IVE ALGOR ITHMS

Data Warehouse

DATABA

SE ADA

PTER

Replay events at high-speed

Test & refine predictive algorithms

Measure & report results and real-time exceptions


R EAL - T IME OPERAT IONAL INTEL L IGENCE C o m p l i m e n t a r y w i t h E x i s t i n g S o l u t i o n s

Sensor Feeds

Data Transmission

Data Collection

Lightweight Agents (listeners)

Existing Archival Database(s)


¤ Dashboards

o  Simple drag-and-drop interface

o  Multiple key performance

indicators on a single panel

o  Real-time, continuous charting

capability on any metric

¤ Visualizations

o  100% web-based

o  Variety of charting types and

customizable alerts

o  Smart-phone access

o  Stunning graphics capabilities

POWERFUL DASHBOARD ING & V ISUAL IZAT IONS


Complementary

o  Doesn’t replace business intelligence solutions, but provides powerful,

complementary operational intelligence capabilities

Massive scalability

o  Ability to provide analytics and alerting on massive volumes of sensor feeds

with minimal infrastructure

Low Total Cost of Ownership (TCO)

o  Minimal infrastructure requirements and rapid time-to-value

o  Utilize existing business intelligence skill sets for real-time data

SUMMARY

S t ream ing Da ta Managemen t


H igh - ve lo c i t y B ig Da ta Ana ly t i c s

Historical queries and data

enrichment

Storing valuable derived streams for future access

Ope

ratio

nal I

ntel

ligen

ce

Logs

Sensors

GPS

Networks

Social media

RFIDs

Servers

Telecom

Smart grid

Oil & Gas

Manufacturing

Logistics

M2M

Telematics

Retail

Internet

Banking

Data centers

Automotive

¤  Continuous Queries over Sliding Time Windows ¤  Analysis and Integration of Unstructured and Structured data ¤  Prescriptive Analytics drives Automated Actions


Rea l - t ime A r c h i t e c t u re Streaming Analysis and Integration for Infinite Flows of Unstructured Data in Real Time

Streaming Agent & Adapter Layer + JDBC API Hadoop Streaming

Query Planner & Optimizer for MPP Execution SQL

Developer Tools

Platform Administration

Streaming SQL Real-time Applications

Real-time Dashboards & Visualization

Logs

Sensors

GPS

Networks Social Media Servers

M2M Telematics

Impala SQL

HBase

HDFS / MR

Hadoop for Stream Persistence, Enrichment & Replay (Optional)

Any external data warehouse, operational system and

enterprise platform


CLEANING & FILTERING

STREAMING ANALYTICS

STREAMING AGGREGATION

CONTINUOUS INTEGRATION

Geospatial Applications Security

Monitoring Real-time

Dashboards Health

Monitoring

QoS and QoE

AN OPERAT IONAL INTEL L IGENCE P LATFORM

Logs Sensors GPS Networks Social Media Servers M2M Telematics


T he SQL s t ream s -S t ream ing P roduc t Po r t fo l i o

s-Server Data Management Platform for Streaming Big Data

s-Analyzer Real-Time Visualization for Streaming

Operational Intelligence

s-Transport Geo-Analytics for Location-based

Applications

s-Visualizer Advanced

Visualization

s-Cloud s-Server EC2 AMI Deployment

s-St

udio

D

evel

oper

& A

dmin

Con

sole

To ta l Co s t o f Pe r fo r mance


R E C O R D S P E R S E C O N D

To ta l Co s t Of Pe r fo r mance ( t o ta l COP ) T h e H i g h - Ve l o c i t y, L ow - L a t e n c y T i p p i n g Po i n t f o r B i g D a t a

Patterns Trends Mining Connections

Searches Inventory Reports Statistics Billing

SOCIAL E-COMM SECURITY TELEMATICS TELECOM

Trading Advertising Alerts Detection Signal

Intelligence

TO

TA

L C

OS

T


Intelligence

TELECOM

Patterns Trends Mining Connections

Searches Inventory Reports Statistics Billing

Trading Advertising Alerts Detection Signal

SOCIAL E-COMM SECURITY TELEMATICS

R E C O R D S P E R S E C O N D

TO

TA

L C

OS

T To ta l Co s t Of Pe r fo r mance ( t o ta l COP ) T h e H i g h - Ve l o c i t y, L ow - L a t e n c y T i p p i n g Po i n t f o r B i g D a t a

Glenn Hout Email | [email protected]

Phone | 650.343.0864

Website | www.sqlstream.com

Upcoming events | www.sqlstream.com/webinars/

Q | A


MACH INE -TO-MACH INE (M2M) APPL ICAT IONS Pe r fo r m a n c e , A n a l y t i c s a n d P r e d i c t i v e H e a l t h M o n i t o r i n g

¤  Transportation

Asset tracking and logistics and consumer Journey Time from GPS data

¤  Telematics

Vehicle health and driver insurance from vehicle sensor data

¤  Healthcare

Remote health monitoring applications from low power micro-sensors

¤  M2M Data Feed Monetization

Streaming data aggregation with subscription-based analytics

ü Improve uptime ü Avoid failures ü One platform


S TREAMING SQL

Example: Compute Average vehicle speed across any subset of network

over rolling time windows from GPS events

- 3 -

In the example shown below, sensor data arriving in the “RoadPositionInfo” stream is used to

calculate average speed per zone over rolling one, five and ten minute windows:

CREATE OR REPLACE VIEW "SpeedZoneStats"

DESCRIPTION ‘Rolling averages for multiple windows partitioned by zone' AS

SELECT STREAM

"zone", -- zone id

"segmentid", -- parent road segment

"speedlimit", -- speed limit for zone

AVG("Speed")OVER last1Min AS "avgSpeed1", -- 1-min running average

AVG("Speed")OVER last5Min AS "avgSpeed5", -- 5-min running average

AVG("Speed")OVER last10Min AS "avgSpeed10" -- 10-mn running average

FROM "RoadPositionInfo"

WINDOW

last1Min AS (PARTITION BY "zone" RANGE INTERVAL '1' MINUTE PRECEDING),

last5Min AS (PARTITION BY "zone" RANGE INTERVAL '5' MINUTE PRECEDING),

last10Min AS (PARTITION BY "zone" RANGE INTERVAL '10' MINUTE PRECEDING);

The output of the application is a continuous stream of exception alerts based on pre-defined

conditions and rolling averages of speed information per road segment over various time windows.

Multiple analytics queries can execute in parallel over any number of different data streams, even

where the arrival rate of each data stream varies.

These simple analytics are used to drive a number of different applications, including real-time

Travel Time predictions, predictive analytics for congestion levels, and real-time alerts for vehicle

speeds in fixed and variable speed limit segments.

LOW LATENCY REQUIREMENTS FOR MULTI-MODAL SOLUTIONS

The case study presented was the first phase of a wider program to deliver a single low latency

platform for all transportation modes. Further real-time monitoring and process automation use

cases have been identified, as well as the integration of traditional data sources, for example:

• Per-segment historical speed/travel-time comparisons, for example, comparisons, in real-time, with the same period yesterday, last week or even last year

• Combine GPS sensor data streams in real-time with roadside camera and signal data, to achieve improved accuracy for congestion and travel time predictions

• Integration with roadside variable speed signs, providing dynamic adjustment of per-segment speed limits in order to respond in real-time to changing traffic conditions

• Driver and user access to the application, providing for example, real-time Travel Time through Smartphones and GPS devices

• Extend Travel Time application to provide end to end journey travel time across multiple transportation modes including heavy vehicles, rail, bus and ferry networks.

36 Copyright © 2012 Proprietary information of SQLstream Inc. All rights reserved