Ultimate Real-Time — Monitor Anything, Update Anything

Ultimate Real-Time Monitor Anything, Update Anything

FME Server Notification Service lets you act on events as they happen, and send information as it becomes available. Publications and Subscriptions let you publish to and monitor different systems and data.

Notifications What they are for

What they are not for

Notifications What they are for

ü  A brief message, usually to trigger an action.

What they are not for x  Transmitting large amounts of

spatial data.

Notifications What they are for

ü  A brief message, usually to trigger an action.

ü  Triggering an FME Server response to an event that happened outside of FME.

What they are not for x  Transmitting large amounts of

spatial data. x  Triggering an FME Server

response to a continuous series of messages (many per second).

Notifications What they are for

ü  A brief message, usually to trigger an action.

ü  Triggering an FME Server response to an event that happened outside of FME.

ü  Sending a message about something that happened in FME.

What they are not for x  Transmitting large amounts of

spatial data. x  Triggering an FME Server

response to a continuous series of messages (many per second).

x  Sending more than one message per second about what’s happening in FME.

FME Server Notification Service

See the list of what built-in publications and subscribers can be: http://fme.ly/protocols

Workflows for “Ultimate Real-Time”

Poll it Push it Update it

Poll it “Are we there yet?”

Polling Workflow 1. Change detection

○  GeoRSS reader ○  Caching ○  Timestamps

2. Run the workspace on a schedule

Timestamp

Scheduling (FME Server)

GeoRSS Reader

GeoRSS Reader

•  Contains change detection logic

•  Option to only read new features

Caching

•  Whole datasets or timestamps

•  ChangeDetector

•  SQL - match on ID

ChangeDetector Example

SQL to find different features: @Value(uuid) = "uuid" AND @Value(aqi) != "aqi"

SQL Example

Timestamps

●  Cache the timestamp

●  Last updated attribute

●  Filter features newer than timestamp

●  Add a timestamp attributes to your dataset

Reads last cached time and replaces with current time

Checks against timestamp and filters out old features

Pros and Cons Caching whole datasets

ü  Easy to set up and detect changes.

x  Need to store entire dataset - processing time?

Caching timestamps

ü No storage needed. x  Harder to set up -

variables, datetimes.

Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?

Story: Real-Time Traffic

✓  Poll Waze feed

✓  Parse XML

✓  Filter

✓  Transform

✓  Validate

✓  Email notifications

Workflows for “Ultimate Real-Time”

Poll it Push it Update it

Push It

Pushing Methods •  Database triggers •  Webhooks

In FME Server:

•  Direct URL •  REST API

Database Triggers

●  Databases can trigger events.

●  Functions with HTTP

requests can push data to FME Server.

Database Triggers

Webhooks

●  i.e. Give systems/services a URL that will respond.

●  Send the notification to

FME Server. ●  Queue the received

notifications and process the requests on a separate thread.

Pros and Cons Database Triggers

ü  Pushes right to FME Server topic.

ü Can push lots of changes to a holding table.

ü  Perfect for real-time database changes.

x  Database permissions.

Webhooks

ü No need to waste effort polling.

ü  Truly real time. x  Complex to set up. x  Not all systems provide

or accept webhooks.

Direct URL Systems can use HTTP to push data to topics or trigger workspaces to run.

Topic POST URL

REST API In-depth interaction

with FME Server

●  Send/receive notifications ●  Run jobs ●  Manage connections ●  Manage users ●  Licensing ●  Backups ●  Scheduling ●  Etc!

Workflows for “Ultimate Real-Time”

Poll it Push it Update it

To Poll or Push?

Poll vs. Push Considerations ●  Simplicity vs. “real-time”ness ●  Can the system push to FME Server? ●  Are jobs too long for polling? ●  API restrictions for polling?

Tip: Use FME Server for internal systems and FME Cloud for external systems.

Update It

Updating Methods

•  FME Writers o  Bulk drop/load/overwrite o  Update option

•  SQLExecuter •  HTTPCaller

FME Writers: UPDATE operation

INSERT, UPDATE or DELETE in the middle of a workflow.

POST data or changes to systems.

Summary

•  Polling workflow: o  Change detection + scheduling

•  Pushing options: o  Database triggers, webhooks o  FME Server: URL or REST API

•  Updating: o  Operations in writer parameters o  SQLExecutor o  HTTPCaller

Questions?

Database triggers tutorial: fme.ly/dbtriggers

Webhooks etc:

fme.ly/web

REST API documentation: fme.ly/restv3