Embed Size (px)
Citation preview
How to Manage Open Police Data – Tips for Data QA/QC
and Automation
Alice Pence, City of AuburnRobert Schultz, RCMP in BC
Cat Gracey, Safe SoftwareDale Lutz, Safe Software
Who are we?
Connect Your Applications | Transform Your Data | Automate Your Workflows
Police Data Initiative 53 U.S. jurisdictions. 41 million people. 90 datasets. And growing!
www.publicsafetydataportal.org
Why Should Police Data be Shared?
● Increase transparency and build trust● Help provide improved public services● Fuel more intelligent policing decisions
The Ethics:
● Need to protect victim identity● Only share when it’s lawful to do so● Must be accurate, free of errors
Dallas Police Department Experience(by the numbers)
12
Dallas Police Department Experience(by the numbers)
45
Dallas Police Department Experience(by the numbers)
67
Alice PenceCity of Auburn
Automating Crime Data to Import into GISAlice Pence GIS/Database SpecialistCity of Auburn, WA
Quick Bio
• BS in Geography from University of Idaho w/Certificate in GIS• In GIS field for 3+ years – been in Auburn, WA for over 1 year
Alice Pence
Our Crimes Database• Stored in a Proprietary Linux system• Raw data with only addresses• Can’t directly dump into GIS- needs to have data transformed for public/internal viewing, QA/QC, a
spatial component to view on maps and applications, attributes renamed…
First impression…
Our Mission
How to view all this data spatially for analysis and reporting?
Our first idea…
A Python ScriptThe Good• It worked. Can add new offenses, change and update as needed.• Ran in windows scheduler to automate.
The Bad• Long, cumbersome script. • Needed multiple databases (.gbd → SDE) to clean and move data.
The Ugly• Can’t easily change connection to Crimes Database, no direct connection to source (connected
through ArcCatalog -then built through Model Builder- then exported out as a Python Script).• No custom alerts or in depth logging capabilities.• Not user friendly for non-python users.
An easier solution?
Our next idea…
FME!!
First ChallengePython FME
This continues on for a few more pages…
How to rename and group 4 letter offense codes
Attribute Mapper Transformer
• As an example: RKBKCrime Type= Felony Crimes Against PersonCrime Name= Robbery
1 2 3
*double bonus: I was able to automatically import codes by a CSV file, AND…FME automatically read in all the source values for me
The Rest of the Process
Into the GIS EnvironmentFME → SDE → MXD → Web Map Service → Online Application = Reporting and Analysis
Wait- there’s more…
Automating the workbench and logging too!
FME Server Set UpEmail notifications when Workbench
succeed or failed
Logging Capabilities (FME Jobs)
Main point: FME rocks
ComparisonPython• Cumbersome/long script• Not very easy to
customize alerts or logs• Hard to edit- other users
might not know the python language
• No friendly user interface
FME• Easy to update codes and
field changes• Other users can easily edit-
even w/minimal knowledge of FME
• Share directly to FME Server for logs and alerts
• Friendly user interface for FME Server and easy to set up
Our Next Great Idea…
Socrata Open Data Portal
Open Data Portal: Crimes Dataset
Updated nightly with new data, thanks to FME Desktop and FME Server
FME Workbench
API Download and Export Tools
Custom Data Lenses (aka Graphs and Maps
Thank you!Alice PenceGIS/Database SpecialistCity of Auburn, [email protected]
Links to sites discussed in slides: Socrata Open Data Portal: https://data.auburnwa.gov/
Auburn Crimes Map:https://maps.auburnwa.gov/html5viewer/Index.html?viewer=crimes
Q&AChat your questions into the GoToWebinar control panel
Robert SchultzRCMP
Property Crime
Spatially Anonymized Property Crime
Property Crime – High Density
Property Crime – Low Density
Open Data in San FranciscoThe new ETL job platform.
Background.
● City’s official open data portal is SF OpenData.
The Vision.
● Empower use of City data.
“Our vision is that the City’s data is understood, documented, and high quality.”
The Project: Increase number and timeliness of datasets
● Few workspace authors● Safe place to try out workspaces
The Results.
● Improved ETL job platform, design and architecture
● 100-500 ETL jobs● Simple ETL Jobs● Use best-practices
View the original presentation by Samuel Valdez and
Janine Heiser
(Presented at the FME World Tour 2016)
Open Data in NYCGIS for public use.
Background.
● The Citywide Street Centerline Database (CSCL) is the authoritative source for location data used by New York City's public safety agencies (NYPD, FDNY) for 911 call-taking and dispatch.
The Vision.
Make CSCL data easily accessible for the day-to-day GIS user.
View the original presentation by Chris Rado
(Presented at the FME World Tour 2016)
The Results.
New version of CSCL created that is better suited for general GIS tasks and made it available to all city agencies, to the public via NYC Open Data and to Batman.
Q&AChat your questions into the GoToWebinar control panel
Thank you!Live chat us at www.safe.com
Free Resources:
FME Desktop Training (June 7-8)
FME Server Training (June 14-15)
Blog post: Guide to Open Data: Using it, Sharing it, and Creating a Portal
Open Data Webinar
Open Data eBook
