Enabling Analytics on Enterprise-Grade IoT Data with … · Hub and Cisco IoT and Data Center technologies Dr. Lars Dannecker, SAP Jens Rannacher, SAP Robert Madl, Cisco Frederic

Enabling Analytics on Enterprise-Grade IoT Data with SAP Data Hub and Cisco IoT and Data Center technologies

Dr. Lars Dannecker, SAP

Jens Rannacher, SAP

Robert Madl, Cisco

Frederic Baudach, Cisco

BRKIOT-2355

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKIOT-2355

• Introduction

• Project background

• Mastering Big Data

• SAP Data Hub Overview

• Containerize Everything

• IoT Reference Architecture

• Cisco Kinetic

• Project Roadmap

Agenda

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5BRKIOT-2355

Introduction

• Dr. Lars Dannecker, SAP

• Development Expert and Big Data Architect

• Jens Rannacher, SAP

• Product Manager SAP Data Hub

• Robert Madl, Cisco

• Global Partner Executive - SAP

• Frederic Baudach, Cisco

• SAP Specialist - Germany

Project Background


01110110100101110100011

0000101101100011010010

111001101101001011000

1000010110110001101101

0101110010011101010101

IoT Train

BRKIOT-2355 7


Scanomat TopBrewer


SAP Data Hub Keynote at TechEd


Smartrail World

10BRKIOT-2355

Mastering Big Data

12INTERNAL© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ

Big Data Changing our Businesses

Accelerating Data Volume Growth

Changing Business Models

22 Billion IoT Devices

by 2018*

4,300% Increase in

Annual Data Production

by 2020**

10% of Orgs will have a

Highly Profitable Business

Unit for Productizing and

Commercializing their

Informational Assets, by

2020****

Shrinking Response Times

80% of New Applications

Using IoT or Machine Data

will Analyze Data in Motion

by 2019***

*IDC: “Worldwide IT Industry 2016 Predictions” Nov 2015

**Forbes: “Big Data Overload: Why Most Companies Can't Deal With The Data Explosion” , Bernard Marr, April 2016

***Gartner, 100 Data and Analytics Predictions through 2020, March 2016

****Gartner, “ Prepare to Monetize Data from Internet of Things” Sept 2016

Digital Transformation Is Driving

The Need for Information

Excellence and High Agility of

Information Delivery


Increasingly Complex Customer Landscapes

Proliferating

Data Lakes

Multiple EDWs,

Data Marts

Enterprise

Applications

ERP, CRM, HR

BI and Data

Visualization

Cloud

Storage

Cloud

Apps

Mobile

Apps

Master Data

Management3rd Party

Data

Data and

systems of

Partners,

Customers


Bring together what belongs together

Cloud

Systems

Social

Networks

Mobile

Devices

Internet

of Things

Big Data

Enterprise

Data

Data WarehouseETL

Intelligent Enterprise Analytics

Sales Order

Master

Data

Customer

Data

ERP Structured

Data

Data Base

Systems

Bridge High Velocity and Batching

Bridge High Volume and High Quality

Bridge Diverse and Structure

Bridge Huge Variety and Trusted

Sources

Bridge Big Data and Enterprise Data


Example: Advanced Fitness Tracker Analysis

Raw Data

10101

$

#123

%&?§

Heart rate, battery

speed, distance

Big Data

#123 10101

Selection, Grouping

Aggregation

Enterprise Data (e.g. ERP)

Transactions

Sales Order

Master Data

Customer

Data

Sales figures

Revenue, Margin

Customer Groups

Location, Details

Intelligent Enterprise

Machine

LearningData

Mining

Advanced

Applications

Intelligent

Assistants


Example: Advanced Fitness Tracker Analysis


New challenges require new technologies

Existing Systems Hadoop/Spark Cloud Storage( i .e. AWS S3)

Machine

Learning(Python, Spark,

Tensorf low)

Containers(Kubernetes, Docker)


Make Big Data Enterprise Ready

Structuring the Unstructured

E2E Landscape Automation

Process data where it resides

Transformers needed - Changing the ETL Principal

Raw Data

Enterprise DWH

10101

$

#123

%&?§

Big Data

#123 10101

Validation

Filtering

Formatting

Data Stream

Search

Parsing

Anonymization

Cleansing

Enrichment

Metadata

Matching

De-normalization

Analytical

Modeling

Selection

Duplicate

Check

Joins / Unions


Transactions are boring… towards flow-based applications


Transactions

Sales Order

Data

Lake

Master Data

Customer

Data


Raw Data

Big Data

Structured & Transaction based Enterprise

Applications

Data & Flow based Scenarios

SAP Data Hub OverviewManage – Orchestrate – Refine


An umbrella for (big) data integration challenges

Existing Systems

Hadoop

Cloud Storage

Machine Learning

Big Data Systems

SAP S/4 HANA

SAP BW/4 HANA

SAP Analytics Cloud

SAP Data Hub

Discovery & Monitoring

Orchestration & Pipelines

Data Management &

Preparation

SAP

HANA

Data Hub

Runtime


SAP Data Hub• Data Pipelines – flow based applications consisting of

reusable and configurable operations, e.g. ETL,

Preparation, Code Execution, Connectors

• Workflows – orchestrate processes across the data

landscape, e.g. executing data pipelines, triggering

SAP BW Process Chains, SAP Data Services Jobs and

many more.

• Governance – metadata repository of information

stored in the connected landscape. Offering discovery,

data quality/integration, profiling and search

capabilities.

Data

Lakes

Cloud Storage EDWs,

Data Marts

Enterprise

Applications

3rd Party

Data & Tools

Cloud

Applications

SAP Data Hub – Key Functions & Capabilities


Connected SystemsSAP Integration & Open Connectivity

SAP Data Hub

SAP Data ServicesData Services Job

Heterogeneous Landscapes

SAP Vora - Containerized SAP Data Hub Pipelines

Serverless infrastructure

SAP HANASDI Flowgraphs

Data Integration into SAP HANA

SAP BWProcess Chains

Data Warehousing Processes Remote

Orchestration

DB Engines

Scheduling

& Monitoring

Data

Pipelines

Access

Policies

Platform Services

Relational Time-Series

Graph Document

Flow-based applications

Custom

Operators

Built-in

ConnectorsScripting

(JS, Python)

Templates

Data Discovery

& Profiling

3rd party and Open SourceDirect Connectivity

Storage, Messaging, APIs

Metadata

Catalog

SAP Data Hub – Whats inside?

SAP Data Hub Adapter

Vora Spark Extensions

UAA SDI

GIT …

Data Hub Application – Data Hub Cockpit

Distributed Runtime

Kubernetes Cluster

Distributed Runtime

Hadoop Cluster


SAP Data Hub – Pipelines rock

SAP BW

SAP Data Services

Data Hub Pipelines

SAP HANA

Scheduling

& Monitoring

Data

Pipelines

Access

Policies

Data Discovery

& Profiling

Metadata

Catalog

Workflow Orchestration

Kafka

SAP Data Services

Data Pipeline

SAP BW

Process ChainSAP Data Hub

Data Pipeline

Change File

Formats

Load Data

into Data Lake

Execute Pipeline

Execute Pipeline

Listen

KAFKA

Write

HDFS

Cleanse

Data

Execute

Python Persist

Result

SAP DATA HUB Application

Data Integration, Streaming, Messaging

Enterprise Applications

Big Data Cluster & Object Stores

SAP DATA HUB Runtime

Data Storage (Hadoop, S3, GCP, Azure, BDS)


Finalize the vision: flow-based big data management

REFINE STRUCTURE MODEL CONSUMECOLLECT

Data Lake Enterprise Data Warehouse

SAP Analytics

Cloud

Cust.

Data

Distributed Big

Data Management

Master

data

IoT

data

Data

prov.

10101

$

#123

%&?§

DemoShopping Center Analysis

Containerize Everything

HOW SERVERLESS COMPUTING

CHANGES THE WAY WE BUILD

APPLICATIONS


DockerContainer vs Virtual Machine

Quelle: docker.com


KubernetesArchitektur


Compose

OpOp

Flow Engine

Flow Editor

Flow Image Composer

Docker Image

Docker Image

Run

Flow Execution

Op Flow Repository

Op

Lib Lib Lib Lib

Op Op Op Op

Lib


Execute

Kubernetes Overlay Network

Kubernetes Runtime

Messaging System

Container Container Container

Container

spawn

run

Container Container


Bring it all together

Flow Engine

Flow Editor

Flow Repository

Op

Lib Lib Lib Lib

Op Op Op

Flow Image Composer

Docker Image

Docker Image

Run

Kubernetes Overlay Network

Kubernetes Runtime

Messaging System

Container Container Container

Container

spawn

run

Container Container

execute

deploybuild

compose

DemoBuild your own content

IoT Reference Architecturepowered by Cisco & SAP


Go from ingestion to consumption with a modern

architecture and ingestion pipeline

• Harness high-volume data results

• Manage single or multiple data lakes and

environments with the needed data quality

• Ingest and process data for enterprise applications

• Automate, design, and run all data processes

• Define and execute the IoT ingestion process

SAP Data Hub – technical use cases

Powerful end-to-end IoT scenario

SAP DATA HUB

Sensors

IoT Gateway and Services

Ingestion and Stream

Big Data Lakes

Data Processing and Orchestration

IoT Applications


SDA

Kubernetes Cluster – Cisco HyperFlex

Devices

SAP VoraSAP Data Hub

Pipeline Engine

Applications

Data Ingestion Pipeline

Op1 Op 2 Op 3 Op 4

Toward a cloud-ready and efficient big data processing Architecture

Spark Custom Code Rules

Cluster Orchestration

Edge / Device Management

Mass Storage

Devices Devices

IoT Service

Message Broker

SAP HANA

Predictive Maintenance

Real-time Demand/Supply Forecast

SPARK

Fraud Detection Brand Sentiment

Product Recommendation

Personalized Care

Processing Engines

Graph Documents

Relational Time Series

REFINE

STRUCTURE

Disk Persistency

HANA Interfaces


Example: Connected Trains

Make Trains Intelligent

Complete working train model

Equipped with Cisco Sensor Arrays

– Cameras: Record images for rail analysis

– Detectors: Identify issues before they occur

– Sensors: Improve economical operations

Additional Network and Analysis Infrastructure

– Wifi Networks: Interconnection of components

– Big Data Analytics: Improve operational control

Next big thing

Make it really intelligent

Combine the power of Cisco hardware with

powerful SAP Software


Example: Connected Trains - Architecture

Architecture

Cisco Kinetic


Companies want to derive value from data

IoT exponentially increases the amount and types of data

BRKIOT-2355 41


App

App

App

App

App

To get value from data

BRKIOT-2355 42


And these things and apps are distributedApp

App

App

App

App

App

AppApp

AppApp

BRKIOT-2355 43


ChallengesApp

App

App

App

App

App

AppApp

AppApp

Complexity of connecting, securing and managing diverse devices

A lot of data remains locked inside its sources

Flexibility to compute data at the edge, data center and/or cloud

No programmatic way to move right data to right apps at right time

No software control to enforce ownership, privacy & security

BRKIOT-2355 44


App

An IoT network fabric is neededAppApp

App

App

App

AppApp

AppApp

Intent-Based Network + Cisco Jasper

Networking

BRKIOT-2355 45


Sensors /

Endpoint

Cloud /

Applications

Gateways

with fog compute

Enterprise

Data Centers

Routing & Switching

with Fog Compute

Cisco IoT network fabric in action

1Fast and secure on-boarding of IoT devices to the network

5Provides real-time visibility and control of connection to each IoT device

3Isolates network traffic of IoT devices with segmentation and prioritization

4Meet environmental needs with portfolio of industrialized network products

2Multiple access methods: Ethernet, WiFi, 3G/4G, LoRa, Mesh, NB-IoT...

Cisco Networking & Cisco Data Center

BRKIOT-2355 46


But challenges remain…

IoT Network Fabric

IoT Data Fabric is needed

Complexity of connecting, securing and managing a set of diverse devices


A lot of data remains locked inside its sources

No programmatic way to move the right data to the right apps at the right time

No software control to enforce ownership, privacy and security

BRKIOT-2355 47


An IoT data fabric is neededApp

App

App

App

App

App

AppApp

AppApp

extracts data, computes data, and moves data

BRKIOT-2355 48


Sensors /

Endpoint

Cloud /

Applications

Gateways

with Fog Compute

Enterprise

Data Centers

Routing & Switching

with Fog Compute

Cisco Kinetic Platform in action

1Automatically

provision

gateways /

endpoints

2Securely extract

and normalize data

3Enforce data privacy,

ownership and

security policies

4Execute the rules

across the system

(biz logic, ops logic)

5Visualization and

intuitive UI

6Securely move

‘right’ data to

right apps

Cisco Kinetic is a system of software running on distributed nodes of edge, data center and cloud. BRKIOT-2355 49


Compute Data Move DataExtract Data

IoT Data Fabric

Cisco Kinetic Platform

Gateway Management Module

Edge & FogProcessing Module

Data Control Module

BRKIOT-2355 50



IoT Data Fabric

Cisco Kinetic + Cisco Networking


Networking

BRKIOT-2355 51



IoT Data Fabric

Cisco IoT


Networking

CiscoIoT Threat Defense

BRKIOT-2355 52


The power of the combination

Complexity of connecting, securing and managing diverse devices


Data remains locked inside its sources

No programmatic way to move right data to right apps at right time

No software control to enforce ownership, privacy & security

Diverse devices are connected, securely and reliably

Compute data at all places where it is valuable

Extracts data and makes it usable

Programmatically moves right data to right app at right time

Executes policies to enforce data ownership, privacy and security

BRKIOT-2355 53


Only Cisco

Trusted Cisco security at both network & software level across the stack

Neutral Data Custodian

Trusted Rule Enforcer

Intent-Based Network +

Cisco Kinetic

for a distributed environment

BRKIOT-2355 54

Project Roadmap


Cisco and Google open hybrid cloud solutionOn Prem/Colo Data Center Google Cloud

Google Cloud Platform

Google Kubernetes Engine

Existing Services

Apps | Data

Private Cloud infrastructure

container platform on hyperconverged

infrastructure

CloudApps

Istio: Hybrid Cloud Service Management

consistent environment

Networking | Security | Private Cloud Infrastructure | Consumption ManagementCSR 1000v, ACI, Stealthwatch Cloud, HyperFlex, Contiv, CloudCenter, AppDynamics

Cisco HyperFlex Systems


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKIOT-2355


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/..

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/

Thank you