Integrating the Enterprise with Event-Driven Messaging – Using FME Server as an ESB

HOW TO USE FME SERVER AS AN

Enterprise Service Bus

WHO WE ARE

Dean HintzSenior Analyst,

Strategic ExpertsSafe Software

Neil HellasEnterprise Solutions Architect

Spatial DNA

Tens of thousands of data pros use FME, the data integration platform with the best support for location data.

Don MurrayPresident,

Safe Software

AGENDA● FME

● Enterprise integration challenges & ESB

● FME Server as ESB

● Spatial DNA examples

● 311 ESB Demo

● Q&A

Technology is always changing

Innovative solutions are possible

New systems are constantly being added

System integration is hard

CHALLENGE

Enables data translation

Integrate virtually any system

Powerful data transformation

No coding required

FME

Connect to Virtually Any Format or Service with FME

400+ SUPPORTED SYSTEMS AND DATA TYPES

Database servers: Oracle, SQL Server, PostGreSQL, MySQL,...

Web systems: MongoDB, CouchDB, Dropbox, Box.com, Amazon S3, OData, Google Sheets, Socrata, Salesforce, Sharepoint, Azure, OGC ...

Business Intelligence: Tableau, Excel, ElasticSearch, Qlik

Extensibility: integration with traditional and emerging systems

● JDBC - often traditional● HTTP - often cloud

Tabular, XML, JSON, CSV, HTML ...

FME TERMINOLOGY

Transformers: Used to connect to systems, extract data, transform and remodel data and then load data. Over 400 available!

Transformers use and produce Features.

Features: Records or rows in spreadsheets and databases or objects in other systems.

Features have Attributes.

Attributes: Fields or columns attached to records, rows and objects.

Three Ways to use FME

FME Server

Automate Deliver Download

** Events / messages: Our main focus here

FME Server Notification Service

●●●

AUTOMATION, EVENTS AND REAL-TIME

Scheduling

REST API

Messaging:

○ JMS○ SQS ○ WebSockets○ WebHooks○ Web Services

API Integration Challenge

Enterprise Integration Challenge

● Multiple, separate applications for different business systems: HR, CRM, BI, ERP, Accounting

● Data silos, duplicate records

● Integration is manual, ad-hoc, point to point

● The more systems that are added, the more difficult it is to manage

Enterprise Integration Challenge: 311 Reporting

Background: Citizens submit reports on the website about problems they notice in the community: missed garbage, potholes, etc. Those reports need to go to the service delivery team who manages related services.

Challenge: The more systems that need to be connected to the 311 database, the more complex the interconnections between databases become.

Specifics: Use topics within the notification service on FME Server as an ESB. All requests between client systems and the 311 database pass through the ESB topics on Server using a common messaging structure. Each system only needs to manage connections to this bus, not other systems.

ESB Demo

Enterprise Integration Challenge

311 DB

EngineeringDB

SafetyDB

Initial architecture

311 DB

EngineeringDB

SafetyDB

TransportDB

Enterprise Integration Challenge

Architecture grows

311 DB

EngineeringDB

SafetyDB

TransportDB

Contact DB

Enterprise Integration Challenge

And grows ...

311 DB

EngineeringDB

SafetyDB

Contact DB

Enterprise Integration Challenge: ESB Approach

More growth = easy

311

ESB

311 DB

EngineeringDB

SafetyDB

TransportDB

Contact DB

Enterprise Integration Challenge: ESB Approach

More growth = easy

311

ESB

Capabilities of an ESB

An ESB can… has…

An ESB can… has…• Route data between systems • Testers, Filters, Notifiers

Capabilities of an ESB

An ESB can… has…• Route data between systems• Transform formats and protocols

• Testers, Notifiers• XML and JSON Templater,

Extractors and HTTPCaller

Capabilities of an ESB

An ESB can… has…• Route data between systems• Transform formats and protocols• Request/Response or

Aynchronous

• Testers, Notifiers• XML and JSON Templater,

Extractors and HTTPCaller• Service Types, JobSubmitter

Capabilities of an ESB

An ESB can… has…• Route data between systems• Transform formats and protocols• Request/Response or

Aynchronous• Publish/Subscribe

• Testers, Notifiers• XML and JSON Templater,

Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications

Capabilities of an ESB

An ESB can… has…• Route data between systems• Transform formats and protocols• Request/Response or

Aynchronous• Publish/Subscribe• Events and messages

• Testers, Notifiers• XML and JSON Templater,

Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols

Capabilities of an ESB

• Route data between systems• Transform formats and protocols• Request/Response or

Aynchronous• Publish/Subscribe• Events and messages• Extensible & pluggable

• Testers, Notifiers• XML and JSON Templater,

Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols• Chaining topics

An ESB can… has…

Capabilities of an ESB

• Route data between systems• Transform formats and protocols• Request/Response or

Aynchronous• Publish/Subscribe• Events and messages• Extensible & pluggable• Publish Metadata

• Testers, Notifiers• XML and JSON Templater,

Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols• Chaining topics• FME Server API

An ESB can… has…

Capabilities of an ESB

Spatial DNA

● ESB Implementation Examples

311 IncidentManagement

ESB Demo

ESB Demo

EngineeringUpdate

SafetyUpdate

311 CaseUpdate

ESB 311Topics

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB Demo

UpdateDepartmentDB.fmw

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

ProcessSafetyMsg.fmw

Update311DB

MessageTransmitter.fmw

ProcessUpdateMessage.fmw

ESB 311Topics

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB Demo

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

ProcessSafetyMsg.fmw

Update311DB

MessageTransmitter.fmw

ProcessUpdateMessage.fmw

UpdateDepartmentDB.fmw

ESB 311Topics

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB DemoESB 311Topics

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

ProcessSafetyMsg.fmw

Update311DB

MessageTransmitter.fmw

ProcessUpdateMessage.fmw

UpdateDepartmentDB.fmw

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

Update311DB

MessageTransmitter.fmw

ProcessSafetyMsg.fmw

ProcessUpdateMessage.fmw

UpdateDepartmentDB.fmw

ESB 311Topics

ESB Demo

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB Demo

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

Update311DB

MessageTransmitter.fmw

ProcessSafetyMsg.fmw

ProcessUpdateMessage.fmw

UpdateDepartmentDB.fmw

ESB 311Topics

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB Demo

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

Update311DB

MessageTransmitter.fmw

ProcessSafetyMsg.fmw

ProcessUpdateMessage.fmw

UpdateDepartmentDB.fmw

ESB 311Topics

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB Demo

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

MessageTransmitter.fmw

ProcessSafetyMsg.fmw

ProcessUpdateMessage.fmw

UpdateDepartmentDB.fmw

ESB 311Topics

Update311DB

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB Demo

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

MessageTransmitter.fmw

ProcessSafetyMsg.fmw

ProcessUpdateMessage.fmw

UpdateDepartmentDB.fmw

ESB 311Topics

Update311DB

311 ESB311PublicMessage

Loader.fmw

311 DB

311MessageDB Summary.fmw

EngineeringDB

ESB Demo

UpdateDepartmentDB.fmw

EngineeringUpdate

SafetyUpdate Safety

DB

311 CaseUpdate

ProcessEngMsg.fmw

ProcessSafetyMsg.fmw

Update311DB

MessageTransmitter.fmw

ProcessUpdateMessage.fmw

ESB 311Topics

ESB 311 Demo: Loader, Update

ESB 311 Demo: Notifications

ESB 311 Demo: Database Summary Report

ESB Configuration on FME Server : Projects

Project Content:● Repository● Workspaces● Topics● Subscriptions● Schedules● Shared resource● DB connections● Web connections● Web socket

ESB Configuration on FME Server : Topics

ESB Configuration on FME Server : Schedules

ESB Configuration on FME Server : DB Connections

311PublicMessageLoader.fmw

To ESB

MessageTransmitter.fmw - Scheduled

To ESB

Update Message via Engineering Topic on 311ESB

{"Status":"Open","Category":"Traffic", "Description":"Flat tire", "Email":"dean@safe.com", "FirstName":"Dean", "ws_topic": "ESB_EngineeringUpdate", "json_featuretype":"ServiceMessage", "Time": "2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC", "Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}

ProcessSafetyMessage.fmw Subscription

From ESB

Update311MessageDB_Safety.fmw Scheduled

To ESB

ProcessUpdateMessage.fmw Subscription

From ESB

Update Message via 311Update Topic

{"Status":"Closed","Category":"Traffic", "Description":"flat tire\nResolved by: Jim", "Email":"dean@safe.com","FirstName":"Dean", "ws_topic":"ESB_311DB_Update", "json_featuretype":"ServiceMessage", "Time":"2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC","Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}

FME Server as ESB: Key Take-aways

● ESB - connection of many systems without increasing architecture complexity

● FME Server’s notification service provides ESB functions and components

● 311 ESB configuration tested using scheduled publication workspaces and

subscription workspaces connected to a set of topics

● Initial ESB design needs to be carefully laid out so the events and message

interactions are clearly mapped

● Define message structure and common data model: key elements, routing

● A good design is easily extendible: start simple, then expand

EnterpriseIntegrationWorkflows

Safe Software

Common Challenges

➔ Field name mapping

➔ Field value mapping

➔ Lookup field mapping

➔ Validating values

➔ Table mapping

➔ Record type mapping

JSON Transformers

Field Names and Values: AttributeManager

Copy from one field to another.

Provide a hard-coded default for a field.

Concatenate multiple field values together into a single field.

Copy part of a value from one field to another.

Value Mapping for Lookup Fields

Validate Values

Summary

● Dilemma of enterprise growth and complexity

● ESB approach / pattern helps manage growth

● FME Server’s notification service as an ESB

● Easy to add new systems and scale capacity without increasing complexity

● Notification service - a great way to integrate FME into your enterprise workflows whether or not you use ESB

Related Webinars● How to Use FME as an Enterprise Integration Platform (Aug 2017)

https://www.safe.com/solutions/enterprise-data-integration

● Web Services & Web Data – Best Practices for Integration (March 2017)

https://www.safe.com/webinars/web-services-fme-2017/

Upcoming:

● Deep Dive into FME Server 2018 (Feb 22, 2018)● FME Server Automations (March 28, 2018)

Resources● Neil’s FME UC 2017 Talk: “Making FME Your ESB”

● FME Server Playground https://playground.fmeserver.com/

● FME Server Notification Docs

● Tutorial: Notifications

● Tutorial: Automation Planning and Deployment

● Tutorial: Reading and Writing JSON

● FME Hub: Custom transformers, templates, coming: Server Projects

ESB Demo project package will be sent to all attendees

Thank you

Dean HintzSenior Analyst,

Strategic ExpertsSafe Software

dean@safe.com

Neil HellasEnterprise Solutions Architect

Spatial DNAneil.hellas@spatialdna.com

Don MurrayPresident,

Safe Software

Q n A