FME WT 2014: (ENG) FME 2014 in action

CONNECT. TRANSFORM. AUTOMATE.

FME World Tour 2014Come Along on a Grand FME Adventure

#FMEWT

March 27 – May 23

Ken BraggEuropean Services ManagerSafe Software


Let’s Go!

Introduction to FME and FME 2014 Database Updating and Deleting Automation with FME Server Advanced Workflows


Why we do what we do. Free your data

Your data shouldn’t be locked into systems, applications, or formats.

Your time is too valuable to be wasted fighting with data.

FME is all about Freedom. Freedom to get the most value from data everywhere.


An explosion of formats! Always more

Always faster.


Key Capability: Data Transformation

▪ Restructure and manipulate data contents

Demo


FME 2014

CONNECT. TRANSFORM. AUTOMATE.Best Practice FME Desktop Training 2012

FormatsFormats


New formats and systems – FME 2014

AIXM 5.x (Aeronautical Information Exchange Model GML version 5.x) (R/W)

Amazon DynamoDB (R/W) Amazon Redshift (Tech Preview) (R/W) Autodesk Revit (R) CARIS Spatial Archive (CSAR) (R/W) Esri ArcGIS Online Feature Service (R) Esri Geodatabase:

(ArcSDE Geodb Raster Catalog) (R) Esri Geodatabase:

(File Geodb Raster Catalog) (R) Golden Software Surfer Binary Grid (R/W) Google BigQuery (Tech Preview) (R/W) Google Cloud SQL Non-spatial (R/W) Google Cloud SQL Spatial (R/W) Google Maps Engine (GME) Tables (R/W)

Google WebP (R/W)

INSPIRE GML (R/W) Landmark Z-Map Grid (R/W) Landmark Z-Map Vector (R) OGP P1/90 Post Plot Positioning Data (R/W) SeabedML (GML) (R/W) Trimble SketchUp (R/W) Virtual Reality Modeling Language

(VRML97) (R) Windows Azure SQL Database Non-spatial

(R/W) Windows Azure SQL Database Spatial (R/W)

CONNECT. TRANSFORM. AUTOMATE.Best Practice FME Desktop Training 2012

TransformTransform


New transformers (highlights).

MapnikRasterizer

WebSocketSender, WebSocketReceiver DatabaseUpdater, DatabaseDeleter GeographicBufferer RasterAspectCalculator, RasterSlopeCalculator S3Uploader, S3Downloader VertexCreator


New transformers.

DatabaseDeleter DatabaseUpdater GeographicBufferer JSONFragmenter MapnikRasterizer NullAttributeMapper PointCloudComponentAdder PointCloudComponentCopier PointCloudComponentKeeper PointCloudComponentRemover PointCloudComponentRenamer PointCloudComponentTypeCoercer PointCloudExtractor PointCloudReplacer PointCloudTransformationApplier RasterAspectCalculator

RasterSlopeCalculator S3Downloader S3Uploader SNSSender SQSReceiver SQSSender VertexCreator WebSocketReceiver WebSocketSender

Basic Demo


Beginners Guide to Database Updating and Deleting


Writing to Databases - Inserts

Easy to INSERT data Automatically INSERTs new data Creates a table if it doesn’t exist Appends to an existing table Simple settings for:

Drop Table or Truncate Table


Writing to Databases - Updating/Deleting

Updating/Deleting existing data isn’t so easy Requires multiple parameter settings Uses format attributes and writer ‘modes’ Might involve multiple transformers Can take some effort to get working

Prerequisites to incremental updates… You need to know which features to

insert / update / delete !


New Transformers

DatabaseUpdater Features enter the transformer and it updates

existing features

DatabaseDeleter Feature(s) enter the transformer and it deletes

existing features.


Formats Supported

Google Fusion Access SQL Server Oracle PostgreSQL/Postgis SQL Lite Windows Azure


Parameters

Attributes to Join on

Where clause option

Columns to Update


Pros

Very easy to configure

Great for relatively simple updates: a few feature types a small number of fields

Great for deleting based on an attribute value


Cautions

No Geodatabase support

No Transactioning or Commit/Rollback handling

Not so useful for updating lots of attributes

Not appropriate for Dynamic Workflows


Other Approaches

Use feature based updates fme_db_operation fme_where

See the KnowledgeBase article: http://

fmepedia.safe.com/articles/How_To/Using-Feature-Based-Attributes-for-Database-updates

http://fmepedia.safe.com/articles/How_To/Using-Feature-Based-Attributes-for-Database-updates




FME Server Automation & Validation


Why Automate?


What is Automation

Event Action


Scheduling

1 Demo


Directory Watch Demo

Automatically convert DWG to Shape and email a link to download

2 Demo


Database Triggers


Email Trigger


Validation Workflows on FME Server

3 Demo


What is Real-Time Data?

Data is delivered for processing as it is collected.

2 Classifications

On-Demand Real-TimeComplex Event ProcessingLow Data Velocity, Low-Mid Data Volume

Notification Based Technology

Continuous Real-TimeContinuous stream of dataHigh Data Velocity, Low-High Data Volume

Dedicated Engine Technology


Real-Time Trigger

Amazon SQS

Amazon SNSWeb Sockets

REST APIJava Messaging

Service(JMS)

UDP

4 Demo

TCP/IP


Workflow Management for Multiple Workspaces


Workflow ManagementWhat are we talking about?


Workflow ManagementWhat are we talking about?

Using the power of FME Workbench to design workflows Moving and Copying Files Data Transformation Quality Control Database Updates Notification (Emails, Mobile Alerts) Etc.


Workflow ManagementWhy do we care?

Because many of you are doing this already! Users have noticed that FME Workbench is a

powerful process control tool Many of you running complex workflows with

multiple workspace already

FME Server Survey 2012 #1 Enhancement Request was “Automated Workflows”


Workflow Management: 2 Case Studies

1. “Call Before you Dig” Client (FME Desktop based)

2. Norwegian Environment Agency (thanks to VAR Norkart)

Please note: Demos are very loosely based on these real projects


Workflow Management - Key Tools

Directory and File Pathnames Reader File Copy Writer

WorkspaceRunner FMEServerJobSubmitter

FMEServerNotifier Notification Service


Case Study #2 Norwegian EA

Goal: automatically add valid data from the SOSI-format to the ArcSDE operational database• geometry must be validated and inside the database before

running any other database updates• split between geometryhandling, attributehandling,

databasehandling• easily add other procedures. Currently working new one to take

attributes from the features and stores these in another database.

View the Application here:http://innsyn.naturbase.no/

http://innsyn.naturbase.no/



SOSI-file

ArcSDE Operational

DB

MSSQL Logging etc

SOSI-file, PDF-report, text-report

1 2 3 4

Verfiy Data and Database updates

Generate paths/published parameters based on attributes in SOSI-file

To Do all checks:- Geometry- SOSI- Attributename/values- Exisiting Ids

Create logfiles (in SOSI, txt, 2D / 3D PDF) and add geometry to ArcSDE

All data verified and all logs updated

To Do:- Wrap-up files from previous FMW-

files into one package and send by e-mail to supplier



Demo 2 – FME Server

Controller Workspace Running two consecutive tasks

1. Quality Control Workspace2. Database Update Workspace

Notification at all stages (Success and Failure)


Where have we been!

Introduction to FME and FME 2014 Database Updating and Deleting Automation with FME Server Advanced Workflows


Thank You for Attending!For more information:

www.fmepedia.com www.safe.com

Contact: [email protected]

FME Partner:

We value your opinion.

http://www.fmepedia.com/

http://www.safe.com/

mailto:[email protected]