ArcGIS GeoEvent Server: Applying Real-Time Analytics€¦ · 2018 Esri European Developer Summit – Presentation, 2018 Esri European Developer Summit, ArcGIS GeoEvent Server: Applying

ArcGIS GeoEvent Server: Applying Real-Time Analytics

Thomas Paschke (Germany) & Daniel Cronin (UK)

AgendaReal-Time DevelopmentWorking with Real-Time DataVisualizing Real-Time DataArcGIS as an IoT PlatformBest Practices & TroubleshootingRoad-Ahead & Summary

1

2

3

4

5

6

Real-Time (& Big Data) Development1

GIS DataWhat has happened, what is happening, what will happen

The ‘current’ snapshot is outdated almost as soon as it’s created…

Credit: iStockphoto/chris_lemmens

Real-Time GIS RequirementsIngestion, Analytics, Notifications and Alerting

Requirement #1

Requirement #2

Continuous Analysis

FeaturesPosition

AlertInsideBoundary

Requirement #3

Applications

Distributed Batch Analysis forlarge amounts of data

ArcGIS Enterprise with real-time & big data GIS capabilities

ingest visualize

analyze store

real-time & big data

millions4K e/s

4K e/s 10Ks e/s

ArcGISEnterprise

spatiotemporalbig data store

GeoAnalyticsServer

IoT Big Data

GeoEventServer

ArcGIS Enterprise with real-time & big data GIS capabilities

ingest visualize

analyze store

real-time & big data

millions4K e/s

4K e/s 10Ks e/s

ArcGISEnterprise


GeoAnalyticsServer

IoT Big Data

GeoEventServer

Distributed Batch Analysis forlarge amounts of data

Working with Real-Time Data2

Ingestion of real-time dataGeoEvent Server: input connectors

GeoEvent Server

Inpu

ts

Out

puts

GeoEvent Services

Poll an ArcGIS Server for Features

Out

of t

he B

ox

Esri

Gal

lery

ActiveMQ

Part

ner G

alle

ry

Amazon IoTAzure IoT

CompassLDEenviroCar

exactEarth AIS

FAA (ASDI)

GNIP

Networkfleet

OSIsoft

Valarm

Waze

AIS

Common Alerting Protocol

Cursor-on-Target

Exploitation Support Data

GTFS

Instagram

Kafka

KML

MQTT

NMEA 0183

RabbitMQ

Sierra Wireless (RAP)

Trimble (TAIP)

Twitter

Poll an external website for GeoJSON, JSON, or XMLHTTP

Receive Features, GeoJSON, JSON, or XML on a REST endpointREST

Receive GeoJSON or JSON on a WebSocketWS

Receive RSS

Receive Text from a TCP or UDP Socket

Subscribe to an external WebSocket for GeoJSON or JSONWS

Watch a Folder for new CSV, GeoJSON, or JSON Files.csv

AIS

GeoEvent Server

Inpu

ts

Out

puts

GeoEvent Services

Dissemination of real-time dataoutput connectors

Out

of t

he B

ox

Add or Update a Feature

Esri

Gal

lery

Amazon IoT

Azure IoT

ActiveMQ

Cursor-on-Target

Hadoop

Kafka

MongoDB

MQTT

RabbitMQ

Twitter

Publish Text to a UDP Socket

Push GeoJSON or JSON to an external WebsiteHTTP

Push GeoJSON or JSON to an external WebSocketWS

Push Text to an external TCP Socket

Send a Text Message

Send an Email

Send an Instant Messageim

Send Features to a Stream Service

Write to a CSV, GeoJSON, or JSON File .csv

Add a Feature to a spatiotemporal big data store

Update a feature in a spatiotemporal big data store

Disseminationconfigure a new output connector by pairing an adapter & transport together

Connector = Adapter + Transport

GeoEvent Server

Inpu

ts

Out

puts

GeoEvent Services

Feature-JSON

Connectors Transport Adapter

GeoJSON

Receive Feature-JSON from Kafka

Receive GeoJSON on a REST endpoint

Feature-JSON

Adapters

GeoJSON

JSON

RSS

Text

XML

CAP

Cursor-on-Target

GeoMessage

Instagram

NMEA

Sierra Wireless (RAP)

Trimble (TAIP)

Twitter

VMF

Esri

Gal

lery

Waze

Feature Service

Transports

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

TCP

UDP

Waze

WebSocket

Esri

Gal

lery Amazon IoT

Azure IoTIRC

Kafka

MQTT

Kafka

HTTP

RabbitMQ

Twitter

you can createyour own

connectors

Extending GeoEventSoftware Development Kit (SDK)

Demo: Ingestion & Visualization

Streaming analyticsGeoEvent Server: geoevent services

input1 output1buffer

geofences

filter(spatial)

input2 output2

geofences

geotagger

output3motioncalculator

geometry inside Zones/.*


• A GeoEvent Service configures the flow of real-time data,- the filtering and geoevent processing steps to perform,- the input(s) where data comes from and the output(s) to which results are sent.

Filters

• Allow event records to pass only if a conditional expression evaluates TRUE

• The expression can use attributes or geometries you have established as geofences

• You can use Boolean logic (AND, OR, NOT) to combine expressions

Spatial Operators

• Spatial operators:

geofences

inside outside

enter exit

intersect disjoint

touches contains

crosses equals

overlaps within

you can createyour own

processors

Streaming analytics, policies & orchestrationGeoEvent Server: processors

GeoEvent Server

Inpu

ts

Out

puts

GeoEvent Services

Buffer Creator

Convex Hull Creator

Difference Creator

Envelope Creator

Field Calculator

Field Enricher

Field Mapper

Geotagger

Incident Detector

Intersector

Projector

Simplifier

Symmetric Difference

Track Gap Detector

Field Reducer Union Creator

Out

of t

he B

oxAdd XYZ

Esri

Gal

lery

Bearing

Ellipse

Event Volume Control

Extent Enricher

Field Grouper

GeoNames Lookup

Range Fan

Reverse Geocoder

Service Area Creator

Symbol Lookup

Track Idle Detector

Unit Converter

Visibility

Motion Calculator Query Report

GeoEvent ServerBeispiel: Konvoy Trennung

Demo: Processing Real-Time Data

Visualizing Real-Time Data3

Visualizationchoosing a service type: stream service, feature service, map service

Stream LayerMap Layer

Feature Layer

• Stream layers in apps subscribe to stream services to immediately visualize observations.- Does not require storage, is low latency, cannot be replayed.

• Map & Features layers in apps periodically poll to visualize most current observations.- Backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS).- History can be retrieved & queried for playback.

desktop

device

web

ArcGISEnterprise

Stream Service

Send Features to a Stream Service subscribe (push)

polling (pull)

Map ServiceFeature Service

Add a Feature to a BDSUpdate a Feature in a BDS


Add or Update a Feature

EGDBGeoEventServer

Stream LayerNow supported in Pro! (2.2 onwards)

Demo: Using StreamLayer in a JS WebApp

GeoEvent Admin REST APIScript based administration

Demo: Using the Admin REST API

ArcGIS as an IoT Platform4

ArcGIS as an IoT Platformenabling geospatial insights with your IoT solution

• An ArcGIS based IoT Platform & Enterprise consists of the following capabilities:

Sensors

Actuators

Devices(or Things)

Env

iro

nmen

t

IoT PlatformEdge Enterprise

Visualization&

Dashboards

Ingestion

StreamingAnalytics

Data Store

BatchAnalytics

Actions &Intelligence

Policy & Orchestration

DeviceManagement

ManagementConsole

- Ingestion

- Streaming Analytics & Policies

- Actions (including Actuation)

- Data Store

- Batch Analytics

- Management Console

- Device Management- Visualization

- Dashboards


Sensors

Actuators

Devices(or Things)

Env

iro

nmen

t

IoT PlatformEdge Enterprise

DeviceManagement

Operations Dashboard for ArcGISInsights for ArcGIS

Esri Story MapsArcGIS Earth

ArcGIS OnlineArcGIS Pro

Collector for ArcGISWeb AppBuilder for ArcGIS

AppStudio for ArcGIS

dashboards

• An ArcGIS based IoT platform consists of the following capabilities:- Ingestion:

- Streaming Analytics & Policies:

- Actions (including Actuation):

- Data Store:

- Device Management: missing functionality in ArcGIS, 3rd party solution must be integrated such as Azure IoT

- Batch Analytics:

- Management Console:

- Visualization:

- Dashboards:


data store

GeoEventServer

ingestion

actions

analytics

GeoAnalyticsServer

analytics

ArcGISEnterprise

policies & orchestration

management console

visualization

GeoEvent server input connectors

GeoEvent Services

GeoEvent output connectors

spatiotemporal big data store

GeoAnalytics Server

Enterprise (Portal & Server Manager)

Map & Feature Services

Operations Dashboard, Insights, AppBuilder


Edge Enterprise

DeviceManagement






dashboards

Sensors

Actuators

Devices(or Things)

Gateways

Env

iro

nmen

t

CoAP

MQTT

AMQP

HTTP

legacy

MQTT, HTTP

IoT Platform

ingestion

Ingestion

StreamingAnalytics

Data Store

BatchAnalytics

Actions &Intelligence


ArcGISEnterprise

GeoEventServer

GeoAnalyticsServer


analytics

policies & orchestration

management console

data store analytics

visualization

ingestion

actions

• Complementing an IoT platform with ArcGIS


Edge Enterprise

Ingestion Actions &Intelligence


DeviceManagement

SpatiotemporalCapabilities

visualization






dashboards

Sensors

Actuators

Devices(or Things)

Gateways

Env

iro

nmen

t

CoAP

MQTT

AMQP

HTTP

legacy

MQTT, HTTP

IoT Platform

ingestion

ingestion

actions

streaminganalytics

data store batchanalytics

• Complementing an IoT platform with ArcGIS

Best Practices & Troubleshooting5

Performance, Resiliency & Scalabilityfactors that influence throughput

Input event counts don’t always tell the whole story

input1 output1buffer

geofences

filter(spatial)

input2 output2

geofences

geotagger

output3motioncalculator



filter(for input1)

filter(for input2)

1000 e/s

2000 e/s

1000 e/s

2000 e/s

2000 e/s

1000 e/s

1000 e/s

1000 e/s

1000 e/s

1000 e/s

1000 e/s

2000 e/s 3000 e/s4000 e/s

Service Design Considerationswhich would you choose?

Service A Service B

Service Design Considerationswhich would you choose?

When possible, pre-filter the input data before ingesting.

Each “branch” in a service contains the same event data.In this example, with three branches, it is creating 3X the volume of data.

Service Design Considerationsnot all components are created equally

A

B

C

Which of these services will process the fastest? Slowest?


A

B

C The first service only contains components that are utilizing the internal service cache, which allows for the fastest processing.


A

B

C The second service utilizes the local ArcGIS Server’s Geometry service to modify the incoming events.These types of requests are typically very quick but can be impacted by geometry complexity.


A

B

C The third service utilizes Network Analyst to return a “drive time” polygon which can significantly impact throughput.

Road Ahead & Summary6

10.6.1 key improvements

• Significant Performance improvements over 10.5.x and 10.6- Configurable multi-threaded writing to the spatiotemporal big data store- Single machine peak throughput velocity up to 10,000 events per seconds (e/s)

• Multi-machine deployments for improved resiliency with linear scalability- Up to 5 machines practically supporting up to 50,000 e/s

• Integrated Public Key Infrastructure (PKI) Authentication (Defense / Intel)

• Improved User Interface for Processor configuration

• ArcGIS Monitor Support

• Stream Layer Support in Pro 2.2 (New render support for published Web Maps)

GeoEvent Server

GeoEvent Server

• Improved User Experience- Embedded Help- Expanded Documentation- Short Training Exercises

• Additional Support for Stream Services- Removing Platform Services Dependency- Dedicated Input Connector- Improved Integration with New Products

• Automatic Configuration Backup to Azure Blob / Amazon S3 bucket

• Quality Focused Improvements

10.7 major epics

Road Ahead content is provided for informational purposes only and is subject to change

Spatiotemporal big data store10.7.x major epics

• Enhance GeoEvent Manager and REST API with ability:- to export data on demand: including Delimited Text or Parquet to S3/Azure Blob- to import data on demand: including Delimited Text or Parquet from S3/Azure Blob

• Z Value Support for Polyline, Polygon, and Multi-Point Geometries

• Additional Symbology Support in ArcGIS Enterprise / Online JavaScript Web Map

Road Ahead content is provided for informational purposes only and is subject to change

• Ingest high velocity real-time data into ArcGIS.

• Perform continuous analytics on events as they are received.

• Store observations in a spatiotemporal big data store.

• Visualize high velocity & volume data:

- as an aggregation- or as discrete features.

• Notify about patterns of interest.

DesktopWeb Device

visualization

live & historicaggregates & features

map & feature servicestream service

live features

ArcGISEnterprise

GeoEventServer


storage

ingestion

analytics

SummaryGeoEvent Server – real-time analytics for your ArcGIS Enterprise

Summaryself-paced training and resources

• Step-by-Step Tutorials, free to download- Introduction- Stream services- Spatiotemporal Big Data Store- Notifications

• Blogs and discussions on the forum- http://links.esri.com/geoevent-forum

• Video recordings of technical workshops- http://www.esri.com/videos

Thank You to Our Sponsors

Documents

ArcGIS GeoEvent Server: Applying Real-Time Analytics€¦ · 2018 Esri European Developer Summit – Presentation, 2018 Esri European Developer Summit, ArcGIS GeoEvent Server: Applying