South Eastern RegionalUsers Conference - 2010

April 26 2010Westin, Charlotte, NC

Pennsylvania Federated GIS

Tom Sweet

RGIS Chesapeake – Wilkes University/

Pennsylvania GIS Consortiumtsweet@pagis.org

8 Site Locations across the USA

Chesapeake, Pennsylvania State UniversityChesapeake, Wilkes University & PaGIS ConsortiumGreat Lakes, University of Wisconsin-MadisonGreat Plains, University of North DakotaMid-South, University of ArkansasPacific Northwest, Central Washington UniversitySouth Georgia, South Georgia Regional Development

CenterTribal Technical Center, Southwestern Indian Polytechnic

Institute

National Consortium for Rural Geospatial Innovations in America

(RGIS)___________________

RGIS Site History?RGIS Site History?

Regional Problem solving:Provide leveraged collaboration

Environmental applications and applied research

Research that supports the design and deployment of locally independent, regionally coordinated, multipurpose Geographic Information Systems

Training, Education and skills retooling

Tech-Transfer

United States Department of Agriculture

Cooperative State Research

Education and Extension Service

Made Possible By___________________

Philadelphia, PA. & Redlands, CA.

Offices of the Environmental Systems Research Institute (ESRI)

jhiggins@esri.com

Made Possible By___________________

Commonwealth of Pennsylvania

Made Possible By___________________

North Central Regional Counter Terrorism Task ForceNorth Central Regional Counter Terrorism Task Force

GovernorGovernor’’s Office of Administration, Bureau of Geos Office of Administration, Bureau of Geo--Spatial TechnologySpatial Technology

Pennsylvania Emergency Management AgencyPennsylvania Emergency Management Agency

North Central Task Force: North Central Task Force: Clinton, Bradford, Lycoming, Potter, Tioga, Sullivan, Clinton, Bradford, Lycoming, Potter, Tioga, Sullivan,

Tioga and Wayne CountiesTioga and Wayne Counties

Pennsylvania Federated Pennsylvania Federated GIS NetworkGIS Network

Access to live Locally Access to live Locally Stored & Maintained Stored & Maintained

Spatial DataSpatial Data

East Central Task Force: East Central Task Force: Berks, Columbia, Luzerne, Montour, Northumberland, Schuylkill Berks, Columbia, Luzerne, Montour, Northumberland, Schuylkill Wyoming, Wayne CountiesWyoming, Wayne Counties

Others:Others: South Central, Northeast, Northwest, PEMA, NGA, others???South Central, Northeast, Northwest, PEMA, NGA, others???

Project history

• The concept of sharing GIS data amongst counties has long been a topic of discussion within the Commonwealth

• That need, to share accurate data rapidly, still exists.

• Several projects in Central Pennsylvania demonstrated that county GIS data could be shared and put into a regional ‘layer’.

A bit more history…

• In subsequent discussions about those early projects, many expressed a desire for this type of data collaboration to be done more in near ‘real time’.

• Rather than just a ‘snap shot in time’

Those ‘wouldn’t it be nice’discussions…

Evolved into what became the initial Federated GIS Pilot Project:

• 2005-2006 Research and development

• 2007 live demonstration of pilot (flooding event)

• 2008-2009 deployment in Counter Terrorism Taskforces

• 2010 – private sector

Key aspects of the Project:

• Each county would build/maintain their data sets to suit local needs and be able to share the data at will without changing their day to day business model

Key aspects continued…

• Data would be able to be shared at regional, state, and federal level without changing the local data structure– And accomplish a sub-goal of not requiring the tax

payer to pay two, three, or more times for the same data

Key aspects continued…

• That data could be shared & combined in near real time with those who needed current data, thus eliminating the issue of outdated GIS data being used for critical decision making from data sets only updated bi-annually or annually

Key aspects continued…

• That a system could be built to provide redundancy of county data and thereby provide continuous access to data during those emergencies where a county’s infrastructure was interrupted or destroyed.

Key aspects worth repeating…

• Certain project components were deemed non-negotiable by participating counties:– Could not change existing current county

workflows;

– Could not change existing data models or standards;

– Maintaining local county independence was absolute;

– Maintaining regional coordination and combining datasets was mandatory.

Typical County Participant?Typical County Participant?

GIS – 1993Rural, 1 – 3 man shopEnterprise GISContributing Data:

Transportation networksHydrologyBuildingsDigital Orthophotos

IMS/SDEAverage relationship with IT (or better)

How did we get here?

IndividualsIndividuals

DepartmentsDepartments

ProfessionalProfessionalProductivityProductivity

InformationInformationManagementManagement

ServicesServices

EnterpriseEnterpriseSystemsSystems

. . . All Provide Value. . . All Provide Value

PA GIS Consortium

Data Distribution Strategy

1993 Model:1993 Model:NonNon--Distributed GIS Distributed GIS (independent and mostly isolated)(independent and mostly isolated)

Local DatabaseLocal Database(e.g. Project GIS)(e.g. Project GIS)

GIS ApplicationsGIS Applications(Project Driven)(Project Driven)

CountyX

MunicipalityY

AgencyZ

DATA

GIS app

DATADATA

GIS app GIS app

InternetMap

Server

Internet

images

BASICBROWSER

Internet

PA GIS Consortium

Data Distribution Strategy

FEDERALACADEMIASTATE

F2 F3 F4

PUBLICLOCALSTATEFEDERAL

UsersUsers

BASICBROWSER

ENHANCEDBROWSER

DataServer

DATADATA

DataServer

DataServer

InternetMap

Server

WEB Server

DATA

Top Down Model: Traditional CentralizedTop Down Model: Traditional CentralizedDistributed GIS (Data Centers)Distributed GIS (Data Centers)

F1

DataServer

DATA

FTP(files)

vectors, featuresimages

Small Scale/Federal Data

Internet/Intranet

PA GIS Consortium

Data Distribution Strategy

LOCALCountyMunicipalityAgencyUniversityOtherL1 L2 L3

PA GIS CONSORTIUMregional coordination

PUBLICOTHER LOCALSTATEFEDERAL

UsersUsers

images

BASICBROWSER

ENHANCEDBROWSER

SDE

ArcIMS

WEB Server

DATADBMS

SDESDE

ArcIMS

WEB Server

DATA

2000 Model:2000 Model:Distributed GISDistributed GIS((locally maintained andlocally maintained andregionally coordinated)regionally coordinated)

vectors, features

Large Scale/Local Data

PA GIS Consortium

Data Distribution Strategy

Portal Model 2002: Enterprise GISPortal Model 2002: Enterprise GIS((locally integrated and maintained)locally integrated and maintained)

ENVIRONM/PLANNING

911

EMERGENCY

TAX

ENGINEERING

SECURE (Data Creator)

FIREWALL

* e.g. JAVA Applet, ArcExplorer

PUBLIC (Data Consumer)

SDE

ArcIMS

WEB Server

DATA

passivepassive

activeactive

GIS Application

ENHANCEDBROWSER

BASICBROWSER

BASICBROWSER

ENHANCEDBROWSER*

Internet/Intranet

FIREWALL

data flowdata flow

UsersUsers

imagesvectors, features

GIS Portals: PA GIS Consortium History

Portal Approach Informs About What's Available

Saves Time, Lowers Redundancy & Enables Sharing

But….(Doesn’t Really Harmonize Information)

It’s a Static Catalog that Involves Dynamic

Integration of Distributed Data

. . . Geographic Inconsistency Along Boundaries

PEMA - Terrorism Task Force: Foundation Philosophy

Develop Regional Counter-Terrorism Task ForcesDevelop county task forces Integrate federal/state/county response Institutionalize mutual aid agreements in the region Establish standing regional response groups Encourage regional networking

……. Challenge and Opportunity. Challenge and Opportunity

GIS Portals Connecting Distributed GIS Services . . .

Accessing Shared Accessing Shared GIS ResourcesGIS ResourcesDatasetsDatasetsData ModelsData ModelsMap ServicesMap ServicesGIS ServicesGIS ServicesApplicationsApplicationsGIS ActivitiesGIS ActivitiesStaff CapacityStaff Capacity

. . . With Tools For Dynamically Integrating Geospatial Servi. . . With Tools For Dynamically Integrating Geospatial Servicesces

Our Challenge

Our Challenge

The road to the PA Federated GIS Project

Step 1Step 1

Commit to collaborateCommit to collaborate

AuthoringAuthoring

Viewing/UsingViewing/Using

PublishingPublishing

Step 2Step 2

Commit to collaborateCommit to collaborate

Enterprise GISEnterprise GIS

Application IntegrationApplication Integration

PublishingPublishing

Step 3 Step 3

Commit to collaborateCommit to collaborate

Data SharingData Sharing

Distributed Data ManagementDistributed Data Management

HarmonizationHarmonization

ReplicationReplication

Application Integration in a COEApplication Integration in a COE

PublishingPublishing

DiscoveryDiscovery

GIS Portals: Connecting the Dots!Connecting Distributed GIS Services . . .Connecting Distributed GIS Services . . .

. . . Analysis on the Fly . . . Analysis on the Fly –– Spatial Data on DemandSpatial Data on Demand

(PaGIS Consortium)(PaGIS Consortium)

Creating the Infrastructure

GIS Portals……………PAGIS Consortium Phase 1

Uniform Harmonization of Data Models And Data …..Common Operating EnvironmentPennsylvania Geospatial Data Sharing Standards (PGDSS)

Periodic Harmonization ……PA Federated GIS Project

Reformatting Existing DataMove towards Community wide data Models

Uniform Data Modeling Uniform Data Modeling Unfortunately we do not have that . . .Unfortunately we do not have that . . .

Uniform Data HarmonizationRequires Change - Transformation

……Time And Leadership Is RequiredTime And Leadership Is Required

TransportationTransportation

CulturalCulturalHydrographyHydrography

Orthophoto BaseOrthophoto Base

•• Agreement On Uniform Data ModelAgreement On Uniform Data Model•• Reformatting Data Reformatting Data •• Changes In WorkflowChanges In Workflow•• Significant Funding. . .Significant Funding. . .

Uniform Data Harmonization - PGDSS

Develop tools, procedures and workflows to test the standards Develop tools, procedures and workflows to test the standards

One Uniform Data Model With Distributed Management Is Powerful

Update Once …… Use Many Times

SupportsSupports Collaborating AgenciesCollaborating Agencies

OneOneWayWay

TwoTwo

WayWayRead /Read /WriteWrite

ViewViewOnlyOnly GIS ServerGIS Server

Read /Read /WriteWrite

NationalNational

StateState

LocalLocal

HierarchyHierarchy

Technically Possible. . . Technically Possible. . . . . . But Requires Institutional Sharing Agreements, Financial S. . . But Requires Institutional Sharing Agreements, Financial Support upport

Periodic Harmonization (Federated) – Pennsylvania Federated GIS

Project

Spatial ETLSpatial ETL•• ExtractingExtracting•• TransformationTransformation

(Conversion)(Conversion)•• Integration / LoadingIntegration / Loading

Database & Server

Data & Services

County Datasets• Road Centerline• Cadastral• Hydrography• Orthophotography

DVD Data Sets

Data Bricks

Using A Standard Data Model

Pennsylvania Federated GIS ProjectHarmonization

Integrating (Transforming) Existing Local, State And Federal Data

CountiesCounties

TaskTaskForcesForces

NationalNationalStatesStates

What started as ….Development of a Regional Collaborative Process

ETLETL ETLETL

Federated GIS Configuration

One Federated GIS Configuration

Federated GIS Configuration: No Proxy Server

Federated GIS Configuration: With Proxy Server

ETL Architecture and approach

ArcGIS Server Interoperability

extensionETL

example

Where are we today? – road ahead

AuthoringAuthoringTierTier

Presentation TierPresentation Tier(Any Viewer)(Any Viewer)

Globe Globe ServicesServices

Map &Map &ChartChart

ServicesServices

ImageImage--ProcessingProcessing

ServicesServices

Open Open Web Web

ServicesServices

TrackingTracking& RSS & RSS

ServicesServicesServing/Serving/

PublishingPublishingTierTier

22--D D ViewersViewers

33--DDViewersViewers

MetadataMetadataCatalog Catalog ServicesServices

GeoGeo--ProcessingProcessing

ServicesServices

COPCOP

3 Tiers3 Tiers

ArcGIS Server

MobileMobile

CountyCounty

Task ForceTask Force

•• Open Open •• InteroperableInteroperable•• Standards BasedStandards Based•• On DemandOn Demand

Pilot Demo

Damage Assessment Tool

Near real time access to local spatial data on a regional level

Near real time access to local spatial data on a regional level

Near real time access to local spatial data on a regional level

Near real time access to local spatial data on a regional level

Summary

• Coordination is Essential– Establishing & leveraging relationships, management & leadership

• Technology – Easier part– Its catching up

• GIS Portals Work ………………………..– Essential to collaboration and discovery

• Data Harmonization Is Possible (Data Models)– Short Term Strategy– A Very Long Term Strategy– Common Operating Environment

• Practical Solutions (ETL ) Exist– For Building On Existing Systems– Immediately Useful