#DataSmartSummit

SUMMIT ON DATA-SMART GOVERNMENT #DataSmartSummit

APPLYING MOBILITY DATA IN CITIES

MICHAEL SCHNUERLEData Officer

City of Louisville@LouDataOfficer

November 2017

Louisville and Waze:Applying Mobility Data in Cities

Michael Schnuerle, Data Officer

Harvard Civic Analytics Network

Summit on Data-Smart Government

November 7, 2017

• Your survey results

• Lots of talking about Waze

• So many slides!

• Smartphone group activity

• Questions and deeper dive

Agenda

Survey Results ● Session participant survey● Civic Analytics Member survey

Harvard Session Survey

Harvard Session Survey

Civic Analytics Network Waze Survey10 Responses

• Allegheny County, PA

• Boston, MA

• Chicago, IL

• Kansas City, MO

• Los Angeles, CA

• New York City, NY

• San Diego, CA

• San Francisco, CA

• San Jose, CA

• South Bend, IN

● Not a Waze Partner● Waze Partner● Storing Data

No one is really using the data

Comments: data is hard to work with, hard to socialize around city, lots of use case ideas (common road backups, emergency management awareness, public visualizations, business development, incident management, historical analysis, real time travel times, street investment impact, city fleet routing, pothole repair, traffic coordination, flow and density, hot-spots, signal effectiveness, event congestion, parking)

Louisville and Innovation● Office of Civic Innovation● Mayor Fischer● Importance of Innovation

Office of Civic Innovation - LouisvilleMayor Greg Fischer Created office in 2010

Grace SimrallChief of Civic Innovation

Ed BlayneyInnovation Project Manager

Michael SchnuerleData Officer

Mary HamptonSenior Data Scientist

Matt Gotth-OlsenInnovation Project Manager

Chris SeidtCivic Technology Manager

Daro Mott Performance Improvement

Data Driven Culture in Louisville

Use data to drive decisions and see opportunities for improvement.

Mayor Greg Fischer at Code for America Summit

Importance of InnovationMost gov employees focused on the daily work. Mayor’s culture encourages breakthrough and innovation work.

Civic Innovation department is run like a start-up:Connect the dots, partner, prove value, find future funding.Help departments work smarter.

Focus Areas:

1. Smart City/Smart Home2. Tech Enabled Public Health3. Social Innovation4. Maker/Hacker Community

Values: Data-driven compassion/ digital inclusion

Waze CCP Overview● Importance of Transit - data, citizens, money, value● CCP Program● Quick Use Cases

• Lots of good data, connect to smart city

• High importance to citizens

• Able to allocate city budget

• Huge value and outcomes possible

• Equity and inclusion opportunities

Importance of Transit for Govs

Data Officer| Bellarmine PresentationOffice of PerformanceImprovement and InnovationOPI2

Waze.com/ccp

• Smartphone navigation app

• Partners with cities across the World with 2-way data sharing - 500+

• We send road closures/construction to open data

• They give us near real-time traffic jams, alerts, problem reports

Connected Citizens Program (CCP)

• Traffic Studies• Signal re-timings, before and after

• Faulty Equipment• Looking at traffic to discover device issues

• Early Analysis• Innovation team can analyze and suggest

Current Waze Project Examples

• Hot Spot Analysis• Evaluation of the most congested corridors

• Building Templates• Putting the right information in the right format

• Real–Time Alerts• Monitoring of a key corridors

Current Waze Project Examples

CCP Data Sharing - from Govs● Road Closure data● USDOT Workzone Standard for Autonomous Vehicles

• Published as open data for all to use

• Updated Daily

• Closures, construction, sidewalks, utility work

• Ingested by Waze for all users

Road Closure and Construction Data

data.louisvilleky.gov/dataset/lane-closures-construction

• Useful for Autonomous Vehicles• Collaborating with other cities and states • Detailed lane and turn details added• Todd A. Peterson, Office of Transportation

Operations, Federal Highway Administration is leading effort.

• Waze is looking at improving this data too.

USDOT Road Closure Workzone Data Standard

CCP Data Sharing - from Waze● Alerts● Jams● Irregularities● Data use agreement

• Near real-time data feed for:• Jams/Congestion (passively collected for users)

• Alerts (actively submitted by users)

• Potholes

• Roadkill

• Cars stopped on road

• Irregularities - speeds on slower roads

What does it give us?

• Missing Signs

• Weather

• Accidents

• Cannot:• share Waze data format spec • share raw Waze data• your CCP data feed URL

• Can:• use and share data internally• share derivative works• share with other CCP partners

Data Use Agreement

Waze Tools● Live Map● Traffic View● App● Future projects

Waze sends us traffic jams, alerts, and user reports. You can see all of that combined on Waze's Live Map: www.waze.com/en-GB/livemap

Everything from Live Map, plus your defined road corridors, unsual traffic. Waze's Traffic View Map: www.waze.com/en-GB/trafficview

Waze App - iOS and Android

Group Activity! ● Let’s Hack Waze!● Waze reports and the data feed

1) Open Waze App - Drive to a Gas Station

2) Report Harvard Roadkill!

3) Watch the Reports Roll In...FYI - Working on citizen defined areas with IFTTT integration

Waze History in Louisville ● Participation● Saving Data● Processing Internally● Hackathon● Processing Externally - Replicate to Other Cities

● Louisville Metro joined in Sept 2015 (5th city)

● Ed Blayney took ownership in Civic Innovation, data saving to Amazon AWS in June 2016

● Worked with IT to set up processing into internal MS SQL database, finished February 2017

● Ed working with Traffic to find more value and benefits. Socializing across city, understanding use cases, providing access/tools/training, defining data structures, merging other data (eg, hackathon)

● Michael Schnuerle building traffic data warehouse with Amazon City on Cloud grant in August 2017, Waze first data set.

● Collaborating with Waze, USDOT, cities for AWS replicable solution.

History of Waze partnership

Use Cases - Maps ● Event analysis● Corridor analysis● Citizen reports

Post Event Analysis

Thunder Over Louisville

Fireworks show to kick off two weeks of Kentucky Derby festivities

Animated Map

Change in Jams after re-timing

Pothole Reports - 311 vs Waze

Waze use is higher on faster roads vs 311 use on slower roads.

Static Map

Animated Map

Use Cases - Traffic Engineers ● Traffic Study Comparison● Hot-spot analysis● Inductive Loop● Other departments

● Traffic department uses Waze data over a time period to do a traffic

study on a corridor, instead of or in addition to a paid study.

● Traffic department monitors changes to a corridor based on tweaks, like

a lightweight traffic study.

● Traffic uses it for hot spot analysis and how to focus their resources.

● Traffic department can detect faulty non-connected equipment at

intersections (eg. induction loops)

● Traffic could use it for post event analysis to check effectiveness of road

closures and routing to optimize future events.

Current Traffic Use Cases

Define geographic corridors, look at jams and irregularities

Top 12 Worst Corridors

Hot Spot Analysis

● Police can monitor a reported road to check for speeders to

deploy officer strategically

● Police can point idle cameras to the location of a reported

collision and alert our Real Time Crime Center for quicker

situational awareness

● Public works can ingest pothole, downed trees, missing sign

reports, can compare to 311 calls

○ Waze use is higher on faster roads vs 311 use on slower roads.

Other Department In-Progress Use Cases

● Real-time alerts and routing for emergency vehicles

● Real-time collision user reports● Vision Zero analysis● Citizen facing notifications in custom

areas

Future Use Cases

911 ReportsGenesis PULSEMASSDOT

Real-time notifications of crashes and road closures

Ability to find crash scenes easier and faster

Best navigation to incidents using Waze SDK

Reducing response times

Waze Event Day

Navigation

New Stadium Game Days - Atlanta Braves

Waze Future - Analytics Tools for CCP

Internal Data Processing● Waze Data Structure ● Power BI● Limitations● FYI: GeoEvent Processor

RAW JSON DATA

FROM WAZE

• Contains descriptions, value ranges, data types for all data

• Available only to CCP partners

Waze Official Specification

Link to file

Internal Database Data Dictionary

• Data Access and visualization tool

• What our traffic department uses to access Waze database and run reports

Microsoft Power BI

Data processed into Louisville internal database

Limitations of Current MSSQL Database

1. Accessible only internally - can’t easily build APIs/services

2. Optimization of indexes = slow

3. No JDBC connections allowed (IT security)

4. Limited to MSSS Management Studio or Power BI access

5. Replication to other cities limited

• _

ESRI GeoEvent Processor: Alerts, actions, real-time

Internal Waze Hackathon● Planning and Setup● Outcomes● Institutionalizing Innovation

• Think technology/data jam session• Except with nerds

• Intense & Time-bound• Typically last for a day or weekend

• Aim for minimum-viable solutions• Not looking for perfect, just good enough

• Even just a good idea will do

What is a hackathon?

Waze Hackathon at LouieLab (city co-working space)

• 9am - Welcome - Ed Blayney, Michael Schnuerle, Mayor Fischer link

• 9:15 - Overview and Technical Details - Michael Schnuerle and Mike Reynolds

• 9:30 -  GIS GeoEvent Processor and Waze/TARC - Andrew McKinney

• 9:40 - Current projects, new ideas, and pitches - Michael S, Ed B

• 10 - Organize into teams based on projects, start working with data - participants

• 12 - BYO Lunch or get Subway

• 2 - Finish projects, start group presentations, discuss successes and roadblocks, decide on future project work - Michael S, Ed B

• 3 - End of event, networking

Waze Hackathon Agenda

Waze Internal : Results• Traffic- Crash Analysis:

• Analysis of significant collisions on Waze

• Police- Cameras for Collisions:

• RTCC/ GeoEvent Processor Integration

• Police- Speeding Monitoring:

• Targeted speed enforcement/review from citizens

• Pollution- Heavy Truck Traffic: 

• Air quality/heavy truck traffic

• Pollution- Getting Cars Off the Road:

• Pushing more people multi-modal uses 

• Rain’s Effect on Driving:

• Driving habits/outcomes during rainfall events

BLOG POST

• Cast a wide net• Invite a variety of experts

• Diversity in thought leads to innovation

• Clear & Consistent Communication• Build a website with resources

• Let people know what to expect

• Get the data ready• Prepping the data allows for more action

• Learn what people want

Tips for hosting your own hackathon

• Strong Facilitation• Participants can get lost in the fray

• Help them stay on track

• Give them the insider knowledge they need

• Follow-up• Hackathons start ideas, they don’t finish them

• Good ideas should be followed up

• Celebrate success

Tips for hosting your own hackathon

Traffic Data Management ● Smart City Framework● Transit Data Warehouse● Amazon AWS● Cloud Benefits and Replication

Smart City Framework

Traffic Data Warehouse• Data Inventory

• Inventory of internal and external agencies transportation data

• Data Warehouse

• Technical build of the Data Warehouse in AWS

• Data Integrations

• Integrations of the various data-sets into the Data Warehouse

AWS City on a Cloud● Louisville won in the “Dream

Big” category this year.● $50K AWS credits for a

Traffic Data Warehouse● Waze is our first data set to

ingest

Waze Cloud Replication ● Collaboration on solution● AWS Architecture● Progress and Future

AWS Solution Collaboration1. Louisville is looking to create an automated AWS Waze

processing solution that can be replicated by any CCP Partner

2. You grab Cloud Formation code and deploy infrastructure stack to AWS

3. Enter your CCP data feed URL4. Analyze, query, extract live and historic data for your city5. We are working to be serverless and inexpensive, but

don’t know costs yet

Working Architecture - 1

Working Architecture - 2

Working Architecture - 3

Current AWS Cloud Formation1. Processes Waze CCP feed

every 2 minutes.2. Takes parameters:

EnvironmentName (dev, test, prod), S3ArtifactsBucket, WazeDataHttpUrl (CCP feed)

3. On Git: github.com/LouisvilleMetro/WazeCCPProcessor

USDOT Secure Data Commons ● ITS JPO Overview● Partnership● SDC Architecture● Informational Session

ABOUT THE ITS JPO DATA PROGRAM

PROBLEM SOLVING APPROACH

•

•

•

•

INVESTING IN A SHARED SOLUTION

SDC STARTING WITH WAZE DATA

Secure Data Commons Overview● Secure Data Commons (SDC) is a new initiative of Intelligent Transportation

System (ITS) Joint Program Office (JPO) from USDOT

● The vision of the ITS JPO Data Program is to increase the agility of the

transportation system to enable technology- and data-driven safety and

mobility gains.

● SDC will make it easy for USDOT and the broader transportation sector to

share and collaborate around sensitive data sets using modern, commercially

available tools that keep up with the pace of change in technology

● Inspired by how the health sector, specifically cancer researchers and

NIH/NCI, are taking on a similar challenge

Secure Data Commons Overview● SDC uses a new approach to provide secure, cloud-based, revocable access to

complex (high volume, velocity, and/or variety) and sensitive (containing private or

proprietary information) datasets in a secure enclave along with analysis tools and/ or

shared computing resources

● The SDC POC is intended as a test case for setting up a laboratory environment

concept for such datasets, and to generate technical, institutional, and lessons learned

that will inform the longer-term ITS JPO data governance strategy as well as other

USDOT efforts in this area

● The intended audience for the POC are researchers (from government, academia, and

industry), modal partners, technical support contractors, and private sector data

sources interested in accessing and using ITS-relevant transportation data that would

otherwise be unavailable

SDC Architecture1. Designed based on

Secure Cloud Computing Architecture (SCCA) from Defense Information Systems Agency (DISA)

2. Sophos provides VPN/firewall/IDS/IPS capabilities acts as a single point of entry and exit)

3. User access is secured through standalone Active Directory installed and configured within SDC Account

4. Connectivity between different VPC’s is through VPN gateways from VDSS to each of the VPC’s

5. Provision workstations for researchers using AWS App Stream

SDC Datalake Logical Architecture

Kibana Metadata Dashboard (development / draft)

Working AWS Architecture Diagram

USDOT Waze Coverage Area (development / draft)

HOW TO GET INVOLVED•••

USDOT vs Waze vs Your Datastore1. USDOT and Waze: similar to a vendor system

a. Good for some use cases and first pass analysisb. Targeted feature set with specific goalc. Rolls out updates based on client requests

2. Your custom cloud/internal datastorea. Extensible, exportable, flexible, customizableb. Detailed analysis, Additional data layer combinationsc. Integration with gov services and systemsd. Needs internal tech skills and time and cost

Conclusion ● Collaboration● Contact Information● Slide Deck LInk

Let’s Work Together!1. Leverage your city’s AWS

experts and collaborate on GitHub with us.

2. github.com/LouisvilleMetro/WazeCCPProcessor

3. Take the code and deploy it to AWS and give us feedback.

Michael Schnuerle

Data Officer

Twitter: @LouDataOfficer

Any Questions?

Contact Info:Michael Schnuerle

Data Officer

michael.schnuerle@louisvilleky.gov

@LouDataOfficer

This Slide Deck:

LSVLL.io/LouisvilleWaze

Some content from Ed Blayney, Innovation PM at OPI -->