Data Validation Victories: Tips for Better Data Quality

  • View
    1.921

  • Download
    2

Embed Size (px)

Text of Data Validation Victories: Tips for Better Data Quality

FME & Data Validation

F M E& D a t aValidationWith Mark Stoakes, George Crowe, and David Ellerbeck

1

Broadcast Room

Made hereS A F E S O F T W A R E

Game RoomMarks Office

2

Colonial Pipeline CompanyFeaturing the

3

Colonial Pipeline Company Business Overview & Safety ShareInterstate common carrier of refined petroleum productsOver 5,500 miles of pipeline stretching from Houston to New YorkHeadquartered in Alpharetta, GATransports approximately 100 million gallons per day: Gasoline Home heating oil Diesel fuel Commercial jet fuel Military fuels

4

Staffing Constraints

Over 5,500 Milesof Linear Assets~ 215 FacilitiesSupported by over 20 Engineering & Design Contracting FirmsInfrastructure Managed with~75 Engineers & Project Mgrs.Asset Data Managed Utilizing6 CAD/GIS SMEs(1 Mgr., 3 GIS, 2 CAD)

5

Standards Validation w/i the AutoCAD Environment

6

Standards Validation w/i the AutoCAD Environment

7

Standards Validation w/i the AutoCAD Environment

8

Standards Validation w/i the FME Environment

9

Standards Validation outside of the AutoCAD EnvironmentAutomationExternal of AutoCAD

ScalabilityPC based and/or Server based

CostTime Reduction

CustomizationPertinent to Industry or Corporate specific standardsEvolve with business processesError logging, correction, and notification

Our solution? FME

10

Reading CAD (DWG) FilesRobust DWG reading capabilitiesObvious: geometry (2D and 3D), layers, line types, etcBlocksBlock AttributesTextExtended entity dataPaper and model spaceInsertions

Extract

11

Manipulation of CAD Objects and EntitiesWorks just like other data typesSpatial DataManipulationComparisonReplacementTabular DataLogical TestsData IntegrationQueries, aggregation, formulasAggregation and reporting

Transform

12

Write to almost any supported data typeBack to CADTo DatabaseTo GISTo

Load

13

Our Approach

StandardsBased

Well Documented CAD StandardRepeatability

Same results every timeEmphasis on Automation

Minimize human interventionReporting Orientation

Identify and report

14

What Does it Mean in FME?Out of the box transformers

Replicated logical tests

Custom transformers (sub-routines)

Standardized reporting approaches

Lets look at it

15

Known Limitations of FME for CAD Data ValidationWriting to multiple Layouts within a single CAD file

Plot Parameters are not supportedFME is not intended to be used for hardcopy output

Your Imagination!

16

Thank You!Full PresentationDavid Ellerbeck, david.ellerbeck@globalinformationsystems.comGeorge Crowe, gcrowe@colpipe.com

Questions?

18Compliance Validation QA/QC- Trust & Verify -

19

The spectrum of complianceSchema / Data ModelAttributesCAD / GIS 2D Geometry3D GeometryTopologyNetworks

Single Item ValidationGeometryValidatorComparative ValidationSpatialRelator

20

Improving Data Compliancy Using FMECity of Kitchener, near Toronto.Centralized GIS in ESRI SDE environment.Maintain ~400 GIS layersUser of FME for last ~8 years.

David van Riel- GIS Technologist -Full Presentation

21

22

23

24

25

26

27

28

CAD Standards Checker

Key transformer:FeatureMergerCAD Standard

29

CAD Standards Checker

Key transformer:StatisticsCalculatorError Reports

30CAD Standards Checker

PDF Output31

Attribute Checker

32

Attribute Checker

Key transformer:Joiner

33

Attribute Checker

34

Topology Checker12 rules:node edge snapnode end snapfloating nodeduplicate nodeline end snapline edge snapcrossed lineline end nodeduplicate linefloating serviceone line two nodesboth line ends snap

35Topology Checker

Key transformers:JoinerNeighbourFinder

36

37

38Digital Submission / Data Compliancy BenefitsTime saving.

Accuracy/completeness of data

Defined deliverables with respect all parties

Everyone is happy!39

40

41

Validation processesAt CCMEO, FME is used for : Feature translationFeature creationFeature validation

Catalog

42

43

44

Validation CatalogISO-19110 compliant

XMLHTMLDatabase

Users

45

Catalog based validationsSpatial relations validationDomain attribute validationProximity validationMinimal dimension validationSegmentation validationData clipping validation

46

Catalog Based Validations - Example - Spatial relations validationBased on Egenhofer-Clementini DE-9IM masks

47

Catalog based validations - Examples - Spatial relations

Attribute value

48

49

50

Catalog validation within FMEReal program example:

51

Automation

52

53

Useful links & resourcesSafe blog The Ultimate Geospatial Data Validation Checklist

More great customer stories at safe.com/eventsFME User ConferenceFME World Tours

Images of these stories54

Questions?

55

Join us for our next webinar

February 25 | safe.com/webinars

56

57