HOW TO USE FME SERVER AS AN
Enterprise Service Bus
WHO WE ARE
Dean HintzSenior Analyst,
Strategic ExpertsSafe Software
Neil HellasEnterprise Solutions Architect
Spatial DNA
Tens of thousands of data pros use FME, the data integration platform with the best support for location data.
Don MurrayPresident,
Safe Software
AGENDA● FME
● Enterprise integration challenges & ESB
● FME Server as ESB
● Spatial DNA examples
● 311 ESB Demo
● Q&A
Technology is always changing
Innovative solutions are possible
New systems are constantly being added
System integration is hard
CHALLENGE
Enables data translation
Integrate virtually any system
Powerful data transformation
No coding required
FME
Connect to Virtually Any Format or Service with FME
400+ SUPPORTED SYSTEMS AND DATA TYPES
Database servers: Oracle, SQL Server, PostGreSQL, MySQL,...
Web systems: MongoDB, CouchDB, Dropbox, Box.com, Amazon S3, OData, Google Sheets, Socrata, Salesforce, Sharepoint, Azure, OGC ...
Business Intelligence: Tableau, Excel, ElasticSearch, Qlik
Extensibility: integration with traditional and emerging systems
● JDBC - often traditional● HTTP - often cloud
Tabular, XML, JSON, CSV, HTML ...
FME TERMINOLOGY
Transformers: Used to connect to systems, extract data, transform and remodel data and then load data. Over 400 available!
Transformers use and produce Features.
Features: Records or rows in spreadsheets and databases or objects in other systems.
Features have Attributes.
Attributes: Fields or columns attached to records, rows and objects.
Three Ways to use FME
FME Server
Automate Deliver Download
** Events / messages: Our main focus here
FME Server Notification Service
●●●
AUTOMATION, EVENTS AND REAL-TIME
Scheduling
REST API
Messaging:
○ JMS○ SQS ○ WebSockets○ WebHooks○ Web Services
API Integration Challenge
Enterprise Integration Challenge
● Multiple, separate applications for different business systems: HR, CRM, BI, ERP, Accounting
● Data silos, duplicate records
● Integration is manual, ad-hoc, point to point
● The more systems that are added, the more difficult it is to manage
Enterprise Integration Challenge: 311 Reporting
Background: Citizens submit reports on the website about problems they notice in the community: missed garbage, potholes, etc. Those reports need to go to the service delivery team who manages related services.
Challenge: The more systems that need to be connected to the 311 database, the more complex the interconnections between databases become.
Specifics: Use topics within the notification service on FME Server as an ESB. All requests between client systems and the 311 database pass through the ESB topics on Server using a common messaging structure. Each system only needs to manage connections to this bus, not other systems.
ESB Demo
Enterprise Integration Challenge
311 DB
EngineeringDB
SafetyDB
Initial architecture
311 DB
EngineeringDB
SafetyDB
TransportDB
Enterprise Integration Challenge
Architecture grows
311 DB
EngineeringDB
SafetyDB
TransportDB
Contact DB
Enterprise Integration Challenge
And grows ...
311 DB
EngineeringDB
SafetyDB
Contact DB
Enterprise Integration Challenge: ESB Approach
More growth = easy
311
ESB
311 DB
EngineeringDB
SafetyDB
TransportDB
Contact DB
Enterprise Integration Challenge: ESB Approach
More growth = easy
311
ESB
Capabilities of an ESB
An ESB can… has…
An ESB can… has…• Route data between systems • Testers, Filters, Notifiers
Capabilities of an ESB
An ESB can… has…• Route data between systems• Transform formats and protocols
• Testers, Notifiers• XML and JSON Templater,
Extractors and HTTPCaller
Capabilities of an ESB
An ESB can… has…• Route data between systems• Transform formats and protocols• Request/Response or
Aynchronous
• Testers, Notifiers• XML and JSON Templater,
Extractors and HTTPCaller• Service Types, JobSubmitter
Capabilities of an ESB
An ESB can… has…• Route data between systems• Transform formats and protocols• Request/Response or
Aynchronous• Publish/Subscribe
• Testers, Notifiers• XML and JSON Templater,
Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications
Capabilities of an ESB
An ESB can… has…• Route data between systems• Transform formats and protocols• Request/Response or
Aynchronous• Publish/Subscribe• Events and messages
• Testers, Notifiers• XML and JSON Templater,
Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols
Capabilities of an ESB
• Route data between systems• Transform formats and protocols• Request/Response or
Aynchronous• Publish/Subscribe• Events and messages• Extensible & pluggable
• Testers, Notifiers• XML and JSON Templater,
Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols• Chaining topics
An ESB can… has…
Capabilities of an ESB
• Route data between systems• Transform formats and protocols• Request/Response or
Aynchronous• Publish/Subscribe• Events and messages• Extensible & pluggable• Publish Metadata
• Testers, Notifiers• XML and JSON Templater,
Extractors and HTTPCaller• Service Types, JobSubmitter• Notifications• Publisher/Subscriber Protocols• Chaining topics• FME Server API
An ESB can… has…
Capabilities of an ESB
Spatial DNA
● ESB Implementation Examples
311 IncidentManagement
ESB Demo
ESB Demo
EngineeringUpdate
SafetyUpdate
311 CaseUpdate
ESB 311Topics
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
ESB Demo
UpdateDepartmentDB.fmw
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
ProcessSafetyMsg.fmw
Update311DB
MessageTransmitter.fmw
ProcessUpdateMessage.fmw
ESB 311Topics
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
ESB Demo
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
ProcessSafetyMsg.fmw
Update311DB
MessageTransmitter.fmw
ProcessUpdateMessage.fmw
UpdateDepartmentDB.fmw
ESB 311Topics
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
ESB DemoESB 311Topics
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
ProcessSafetyMsg.fmw
Update311DB
MessageTransmitter.fmw
ProcessUpdateMessage.fmw
UpdateDepartmentDB.fmw
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
Update311DB
MessageTransmitter.fmw
ProcessSafetyMsg.fmw
ProcessUpdateMessage.fmw
UpdateDepartmentDB.fmw
ESB 311Topics
ESB Demo
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
ESB Demo
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
Update311DB
MessageTransmitter.fmw
ProcessSafetyMsg.fmw
ProcessUpdateMessage.fmw
UpdateDepartmentDB.fmw
ESB 311Topics
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
ESB Demo
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
Update311DB
MessageTransmitter.fmw
ProcessSafetyMsg.fmw
ProcessUpdateMessage.fmw
UpdateDepartmentDB.fmw
ESB 311Topics
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
ESB Demo
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
MessageTransmitter.fmw
ProcessSafetyMsg.fmw
ProcessUpdateMessage.fmw
UpdateDepartmentDB.fmw
ESB 311Topics
Update311DB
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
ESB Demo
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
MessageTransmitter.fmw
ProcessSafetyMsg.fmw
ProcessUpdateMessage.fmw
UpdateDepartmentDB.fmw
ESB 311Topics
Update311DB
311 ESB311PublicMessage
Loader.fmw
311 DB
311MessageDB Summary.fmw
EngineeringDB
ESB Demo
UpdateDepartmentDB.fmw
EngineeringUpdate
SafetyUpdate Safety
DB
311 CaseUpdate
ProcessEngMsg.fmw
ProcessSafetyMsg.fmw
Update311DB
MessageTransmitter.fmw
ProcessUpdateMessage.fmw
ESB 311Topics
ESB 311 Demo: Loader, Update
ESB 311 Demo: Notifications
ESB 311 Demo: Database Summary Report
ESB Configuration on FME Server : Projects
Project Content:● Repository● Workspaces● Topics● Subscriptions● Schedules● Shared resource● DB connections● Web connections● Web socket
ESB Configuration on FME Server : Topics
ESB Configuration on FME Server : Schedules
ESB Configuration on FME Server : DB Connections
311PublicMessageLoader.fmw
To ESB
MessageTransmitter.fmw - Scheduled
To ESB
Update Message via Engineering Topic on 311ESB
{"Status":"Open","Category":"Traffic", "Description":"Flat tire", "Email":"[email protected]", "FirstName":"Dean", "ws_topic": "ESB_EngineeringUpdate", "json_featuretype":"ServiceMessage", "Time": "2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC", "Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}
ProcessSafetyMessage.fmw Subscription
From ESB
Update311MessageDB_Safety.fmw Scheduled
To ESB
ProcessUpdateMessage.fmw Subscription
From ESB
Update Message via 311Update Topic
{"Status":"Closed","Category":"Traffic", "Description":"flat tire\nResolved by: Jim", "Email":"[email protected]","FirstName":"Dean", "ws_topic":"ESB_311DB_Update", "json_featuretype":"ServiceMessage", "Time":"2017-12-01T07:30:01","City":"Vancouver", "Source":"web form","Province":"BC","Date":"2017-12-01", "Department":"Engineering", "Street":"123 4th st", "LastName":"Hintz", "MessageID":"70f4c197-d669-11e7-97b9-161c33569574"}
FME Server as ESB: Key Take-aways
● ESB - connection of many systems without increasing architecture complexity
● FME Server’s notification service provides ESB functions and components
● 311 ESB configuration tested using scheduled publication workspaces and
subscription workspaces connected to a set of topics
● Initial ESB design needs to be carefully laid out so the events and message
interactions are clearly mapped
● Define message structure and common data model: key elements, routing
● A good design is easily extendible: start simple, then expand
EnterpriseIntegrationWorkflows
Safe Software
Common Challenges
➔ Field name mapping
➔ Field value mapping
➔ Lookup field mapping
➔ Validating values
➔ Table mapping
➔ Record type mapping
JSON Transformers
Field Names and Values: AttributeManager
Copy from one field to another.
Provide a hard-coded default for a field.
Concatenate multiple field values together into a single field.
Copy part of a value from one field to another.
Value Mapping for Lookup Fields
Validate Values
Summary
● Dilemma of enterprise growth and complexity
● ESB approach / pattern helps manage growth
● FME Server’s notification service as an ESB
● Easy to add new systems and scale capacity without increasing complexity
● Notification service - a great way to integrate FME into your enterprise workflows whether or not you use ESB
Related Webinars● How to Use FME as an Enterprise Integration Platform (Aug 2017)
https://www.safe.com/solutions/enterprise-data-integration
● Web Services & Web Data – Best Practices for Integration (March 2017)
https://www.safe.com/webinars/web-services-fme-2017/
Upcoming:
● Deep Dive into FME Server 2018 (Feb 22, 2018)● FME Server Automations (March 28, 2018)
Resources● Neil’s FME UC 2017 Talk: “Making FME Your ESB”
● FME Server Playground https://playground.fmeserver.com/
● FME Server Notification Docs
● Tutorial: Notifications
● Tutorial: Automation Planning and Deployment
● Tutorial: Reading and Writing JSON
● FME Hub: Custom transformers, templates, coming: Server Projects
ESB Demo project package will be sent to all attendees
Thank you
Dean HintzSenior Analyst,
Strategic ExpertsSafe Software
Neil HellasEnterprise Solutions Architect
Spatial [email protected]
Don MurrayPresident,
Safe Software
Q n A