Embed Size (px)
Citation preview
How to Optimize GIS Workflows
With Mark Stoakes & Steve MacCabe
Who are we anyway?
Connect Virtually Any Format with FME
FME & GIS can do amazing things
Here is a taste of what’s possible.
Manage your attributes
Validate your attributes
Distribute a Validation Report
Automate the process
Managing Attributes
Attribute ManagementWhy another attribute handling transformer?
➢ Because there are already too many… plus we needed a couple other operations
Goal: consolidate the attribute management transformers
Doomed transformers:AttributeCreatorAttributeCopierAttributeRenamerExpressionEvaluatorStringConcatenatorAttributeKeeperAttributeRemover
Plus… …many string operations:case changetrim/padsubstring/regex
One transformer to rule all your attributes
● Create attributes
● Rename attributes
● Copy attributes
● Remove attributes
● Set attribute values
● Re-order attributes
● Concatenate attributes (Text Editor)
● Arithmetic expressions (Arithmetic Editor)
● Set conditional values (Conditional Value)
● Reference earlier attributes
AttributeManager
AttributeManager creates an opportunity for more focused and simplified attribute management.
#ThanksFME2016
Let’s Go - Demo!
AttributeManger:
Lets you organize your attributes in
your workflows
Attribute Validation
Attribute Validation – why?
If you distribute poor quality data then you've amplified the data issue.
If you distribute good quality data you've amplified the benefits.
Data validation is key to many FME workflows…
Attribute Validation – Standards
Bunch of ISO standards – ISO19xxxMostly replaced by - ISO 19157
Also CAD & Data Model Standards
Attribute Validation – FME ToolsAttribute Validation – Key Tools
● Tester and TestFilter● AttributeCreator (conditional tests)● AttributeClassifier● AttributeRangeFilter & AttributeFilter● NullAttributeMapper● DuplicateRemover● StringSearcher● StatisticsCalculator
AttributeValidator...
Goals:Simplify validation workflows (Reduce number of transformers)
Be more confident about the data you’re loading into a destination dataset
...say goodbye to bad data
#ThanksFME2016
Attribute ValidationSome of the validation tests available:
➢ Attribute type (integer, float, char, xml, json, etc.)➢ In – either a list or range – good for domain validation➢ Regular Expressions➢ Unique➢ Not Null➢ …and more…
Attribute Validation Output Output is:
➢ Summary Attribute:_fme_validation_message
➢ List of all messages: _fme_validation_message_list{}
Attribute 'CodePrefix' with value 'ABE' fails check for Matches Regular Expression '[ABCD]{3}'{0} Attribute 'CodePrefix' with value 'ABE' fails check for Matches Regular Expression '[ABCD]{3}'{1} Attribute 'num_measures' with value '12' fails check for in Range '[0,10]'{2} Attribute 'CodePrefix' with value 'ABE' fails check for in 'ABC,ABD,TXU,TXV'
Let’s Go - Demo!
AttributeValidator:
Lets you scrutinize your attributes and
simplify your data validation
workflows
Q&AChat your questions to us
Writing Features...a different approach
Traditional vs. state-of-the-art...Write data with a writer
Write data with a FeatureWriter
Write the conclusion to your data process
Write data in the middle of workflows to avoid chaining workspaces.
Write the summary of your data output.
#ThanksFME2016
Use the FeatureWriter to allow post-processing freedom.
#FreeYourData
Some Easy Wins:• Create summary reports
– Email an attachment• Copy data after writing
– FTP results– Load into Dropbox
Automation Workflows• Quality checking• Database operations• Database validation• FME Server Notification
Transformation
...transform your data
A new pattern for database workflows
Transform your database workflows
SQLCreator
...drop constraints
FeatureReader
...read your data
Transformation
...transform your data
FeatureWriter
...write to database
SQLExecutor
...enable constraints
SQLExecutor
...validate/index/unique
A new pattern for FME Server notification workflows
FeatureReader & FeatureWriter simplies you workspace
Let’s Go - Demo!
FeatureWriter:
Lets you post-process your
output data
Automation
FME Server OverviewFME Engine workhorse and runs FME Workspaces
Automate Deliver Data Download Data
Love
FME!
ScaleReal Time Data
Let FME Server do the work
Examples of things we can automate:● Directory watcher● Run the workspace through a notification ● Data Delivery
○ Dropbox○ email
Let’s Go - Demo!
Summary
Take advantage of our new transformers.
Free StuffThere are too many to list so be sure to check out www.safe.com
for more.
Knowledge Articles● AttributeManager● AttributeValidator
Blog Posts● FeatureWriter● AttributeManager
Free FME Training● FME Desktop Basic: March 8-9● FME Server: March 15-16● FME Desktop Advanced: March 22-23
60+ CITIESApril - J une
Poll Question
Which of today's tasks do you see yourself performing on your GIS data?
Q&AChat your questions into us
Thank you!
Get a free trial of FME at www.safe.com
Resources: Attribute ValidationOriginal PR: 8945Knowledge Center: Validate your Data's Attributes with the AttributeValidator Transformer
Webinar 2015: Data Validation Victories: Tips for Better Data Quality:
Data Validation & Quality Assurance
WWTour:FME & Data Validation: http://www.slideshare.net/SafeSoftware/fme-data-validation
WWTour 2014: Validation and Quality Control with FME & FME Server and also at SlideShare
Validating your data before loading it into a database: video by Robyn
FME 2014 UC - Colonial Pipeline: CAD Data Validation using FME
Automated Quality Control: ISo 9000:2000 Quality Measures ISO 19157/19131 Metria 2014
Berk Bayer 2014 Application of FME to Maintain Quality Control (QC) over Standardized Spatial Data
FME WWTour 2013 – Introducing Geometry Validator
Resources: AttributeManagerKnowledge Center: Control Your Data's Attributes with the AttributeManager Transformer
