Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Harnessing Data-in-Motion with Hortonworks DataFlow

Edge intelligence for IoT with Apache MiNiFi (subproject of Apache NiFi)

Joe PercivallSoftware Engineer

Anna YongProduct Marketing


Agenda• Apache NiFi Refresher

• Intro to Apache MiNiFi

• Demo

• Next Step

• How MiNiFi can be used – use cases


EMBRACE AN OPEN APPROACH

MASTER THE VALUE OF DATA

EVERY BUSINESS IS A DATA BUSINESS


Constrained High-latency Localized context

Hybrid – cloud / on-premises Low-latency Global context

CoreInfrastructure

Harnessing Data in Motion

RegionalInfrastructureSources


Hortonworks DataFlow Manages Data in Motion

CoreInfrastructureSources

Constrained High-latency Localized context

Hybrid – cloud / on-premises Low-latency Global context

RegionalInfrastructure


Requirements for Data in Motion

Perishable Insights

ConnectivityPrioritization

Security

AdaptabilityExtensibility

Scalability

Provenance

Real-Time


Connecting Data Between Ecosystems Without Coding: 170+ Processors

Hash

Extract

Merge

Duplicate

Scan

GeoEnrich

Replace

ConvertSplit

Translate

Route Content

Route Context

Route Text

Control Rate

Distribute Load

Generate Table Fetch

Jolt Transform JSON

Prioritized Delivery

Encrypt

Tail

Evaluate

Execute

HL7

FTP

UDP

XML

SFTP

HTTP

Syslog

Email

HTML

Image

AMQP

MQTT

All Apache project logos are trademarks of the ASF and the respective projects.

Fetch

mailto:[email protected]


Apache NiFi

Guaranteed delivery Data buffering

‒ Backpressure‒ Pressure release

Prioritized queuing Flow specific QoS

‒ Latency vs. throughput‒ Loss tolerance

Data provenance

Recovery / recording a rolling log of fine-grained history

Designed for extension Visual command and control Flow templates Policy based security Clustering

Key Features




• Demo

• Next Step



Simplified Ingestion ExampleLet’s consider the needs of a courier service

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster

Distribution Center Core Data Center at HQ

Server Cluster

On Delivery Routes

Trucks Deliverers

Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/

https://thenounproject.com/creativestall/


https://thenounproject.com/rigo/

https://thenounproject.com/bdesign.by/

https://thenounproject.com/epearson001/


Courier service from the perspective of NiFi

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster


Server Cluster

Trucks Deliverers


NiFi NiFi NiFi NiFi NiFi NiFi

On Delivery Routes











Courier service from the perspective of NiFi & MiNiFi

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster


Server Cluster

Trucks Deliverers


Client Libraries

Client Libraries

MiNiFi

MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi

Client Libraries

On Delivery Routes











Created to more effectively collect data at the edge


Apache NiFi





Data provenance


Designed for extension Visual command and control Flow templates Policy based security Clustering

Key Features






Data provenance

Different from Apache NiFi Design and Deploy Warm re-deploys

Key Features

Apache NiFi MiNiFi


Designed for extension


Command and Controlvs.

Design and Deploy


Different MiNiFi Distributions

Java– <40MB binary distribution– Requires Java 1.8– More feature complete– Targeted for any systems that can run a JVM (ie. Servers, Raspberry Pi)

C++– 600KB code size and static data ~50KB– Dynamic heap of ~1MB based on use-case– Targeted for resource constrained environments (ie. edge IoT devices)

Both use same config format and use NiFi terminology

Different focuses depending on requirements


Typical Simple ExampleTail a rolling file -> Site to Site


Typical Simple ExampleTail a rolling file -> Site to Site


Simple Config.ymlTail a rolling file -> Site to Site


NiFi and MiNiFi, JAVA and C++


NiFi vs MiNiFi Java Processes

NiFi Framework

Components

MiNiFi

NiFi Framework

User Interface

Components

NiFi


NiFi Java Processes

Bootstrap

NiFi

UI

bootstrap.conf

nifi.properties

flow.xml.gzreads &modifies

reads

reads

starts

NiFi MiNiFi


MiNiFi Java Processes

MiNiFi

Bootstrap

ConfigurationChange Notifier(s)

bootstrap.conf

nifi.properties

flow.xml.gzreads

reads

starts

config.ymltransforms

reads

into

NiFi MiNiFi


Same Extensible framework (nars)

NiFi standard processors are bundled– TailLog– UpdateAttribute– Route on content and attributes– PutEmail– ….

Allows MiNiFi Java to use NiFi processors


MiNiFi C++ Features Same design and deploy configuration Initial set of processors

– TailFile– GetFile– GenerateFlowFile– LogAttribute– ListenSyslog

Site to Site Client implementation in C++ for talking to NiFi instances




• Demo

• Next Step



Simple Config.ymlTail a rolling file -> Site to Site


Courier service from the perspective of NiFi & MiNiFi

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster


Server Cluster

Trucks Deliverers


Client Libraries

Client Libraries

MiNiFi

MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi

Client Libraries

On Delivery Routes











Demo: Digital Temp & Humidity Sensor, Raspberry Pi Zero, MiNiFi

Raspberry Pi Zero

with MiNiFi(link) USB hub for

power & WiFi

Digital Temp & Humidity Measurement Sensor

https://www.raspberrypi.org/products/pi-zero/




• Demo

• Next Step



MiNiFi Command and Control

Design Flow at a centralized place, deploy on the edge Version control of flows

– Align with NiFi SDLC work

Agent status monitoring Bi-directional command and control

Currently a feature proposal, initial version being architected

https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control






• Demo

• Next Step



CoreInfrastructure

Hortonworks DataFlow Use CasesRegional

InfrastructureSources

Dataflow Management• On-ramp into Hadoop• Log Collection / Splunk Optimization• Cyber Security• IoT Ingestion• Deliver data into stream processing engines

Real-time Event Processing (Kafka, Storm)

Move data between from on-prem and cloud environments


Apache MiNiFi Use Case: Distributed Systems: Retail POS

100’s or 1000’s or more terminals to gather: Real-time pricing info Real-time inventory updates Real-time offers Traceable logs of transactions Log of user-interactions for trouble-shooting

devices Predictive help desk

Apache MiNiFi Use Case: Distributed Systems: Connected Car

MiNiFi

SD Card

Wi-Fi

DSRC

LTE

Raw data via CAN, Ethernet

Car Wireless Gateway

Buffer offline delay-tolerant data

1

Real-time data

1

4

Wi-Fi AP

DSRC RSU

eNodeB

5

5

Possibly modify route or adjust sampling rate and prepare to flush delay-tolerant buffers when SDRC RSU or Wi-Fi AP in range

2

Data Storage

Decision-Making Server

Map Database

3

Hortonworks maximizes intelligence with platforms that deliver actionable insights from connected car data


GENIVI Alliance: Open Source, In-Vehicle Infotainment Software

The GENIVI Alliance is a nonprofit industry alliance committed to driving the broad adoption of specified, open source, In-Vehicle Infotainment software.

https://www.genivi.org/about-genivi

https://www.genivi.org/about-genivi


Apache MiNiFi Use Cases: Distributed Systems of Devices

Routers Security cameras Cable modems ATM

Fleet of Trucks Manufacturing Line Security Appliance Point of Sale

Weather detection system Thermostats Utility/Power meters Fleet of Ships

Any distributed system of devices with data to be collected


Questions?


Learn more and join us!

Apache NiFi sitehttp://nifi.apache.org

Subproject MiNiFi sitehttp://nifi.apache.org/minifi/

Subscribe to and collaborate [email protected]@nifi.apache.org

Submit Ideas or Issueshttps://issues.apache.org/jira/browse/NIFIhttps://issues.apache.org/jira/browse/MINIFI

Follow us on Twitter@apachenifi

http://nifi.apache.org/

http://nifi.apache.org/minifi/

http://nifi.apache.org/minifi/



https://issues.apache.org/jira/browse/NIFI

https://issues.apache.org/jira/browse/MINIFI

https://issues.apache.org/jira/browse/MINIFI


Questions?

Hortonworks Community Connection:Data Ingestion and Streaminghttps://community.hortonworks.com/


Thank you!

Technology

Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi