Embed Size (px)
Citation preview
ARC’s Project Evaluation Framework
AMPO ConferenceOctober 19, 2017
David D’Onofrio Principal Planner
Atlanta Regional [email protected]
470.378.1559
Outline
Performance Based Planning
ARC’s Project Evaluation Task Force
Project Evaluation Decision-Making Framework
Outcomes
Lessons Learned
Performance Based Planning
ARC has a rich history of performance-based regional planning
Focused in 2016 on translating federal/state/regional goals to performance measures and metrics for use in TIP project solicitations
Incorporated SHRP2 C02 framework into a 6-month TIP Project Evaluation Taskforce
Major Policy Guidance
2008
2013
20142010
2008
2009
2015
2008
2015
2007 2011
2015
N/A
2008
2008
2014
2011
2007
2009
Comprehensive Transportation Plans (CTPs)
Transportation Environment Economics Community Cost
Mobility Ecosystems/Habitat Economic Impact Land Use Cost
Reliability Water QualityEconomic Development
Cultural & ArcheologicalResources
Cost-Effectiveness
Accessibility Wetlands Social
Safety Air Quality Environmental
Climate Change
Environmental Health
ARC has addressed the blue items in past performance measures
SHRP2 – C02 Performance Measures
The Atlanta Region’s Plan Policy Goals
Key Decision Points (KDP) Framework
Points along the planning process where the evaluation of funding applications occurs
Three KDPs in the Atlanta process:
KDP 1 – Policy Filters
KDP 2 – Project Evaluation
KDP 3 – Final Factors & Project Selection
KDP1 – Policy Filters
KDP2 – Technical Evaluation
ARC evaluates all projects submitted on technical grounds
Projects are broken out into distinct “project types” and scored compared to similar projects
Criteria, measures and metrics were established as part of ARC’s Project Evaluation Taskforce in 2016
Projects are placed into tiers based on a combination of performance and cost-effectiveness measures
Performance Criteria BikeRoad
ExpansionEtc…
Wo
rld
Cla
ss
Infr
astr
uct
ure
Mobility & Congestion
Reliability
Network Connectivity
Multimodalism
Asset Management & Resiliency
He
alth
y Li
vab
le
Co
mm
un
itie
s
Air Quality & Climate Change
Safety
Cultural/Environmental Resources
Social Equity
Land Use Compatibility
Competitive Economy
Goods Movement
Employment Accessibility
Cost Cost Effectiveness or Benefit-Cost
KDP2 Project Evaluation Matrix
KDP2 – Tools & Data
Activity Based Travel Model / VISUM Modeling CMAQ Calculator / AREES Model / MOVES Emissions Model Real-World Megadata (INRIX/HERE) Walk! Bike! Thrive! GIS data Regional Priority Networks GIS files Regional UGPM GIS files SHRP2 Ecological GIS data FHWA Safety Countermeasures Database / GEARS Safety Data GDOT Traffic Counts and Asset Management Data Project Sponsor Data
KDP2 – Weights
Criteria Bike/Ped/Trail
Roadway Asset
Management
Roadway
Expansion& TSM&O
TransitExpansion
TransitAsset
Management1
Asset Management & Resiliency
- 14.9 % - -22.0 % / 19.2 %
Mobility&
Congestion13.7 % 13.8 % 13.0 % 13.5 % -
Safety 14.5 % 14.4 % 13.4 % 8.5 %22.0 % / 19.2 %
Network Connectivity 14.4 % 12.9 % 12.4 % 13.5 % -
Reliability - - 12.1 % 12.0 % -
Multimodalism 12.6 % 11.8 % 11.3 % 10.2 % -
Employment Accessibility
10.4 % 10.2 % 10.3 % 11.6 %24.3 % / 21.2 %
Land Use Compatibility
11.5 % - - 10.5 % -
Social Equity 9.7 % 8.3 % 7.0 % 9.5 %20.8 % / 18.2 %
Air Quality & Climate Change
6.3 % - 7.3 % 6.5 %0.0 % / 12.6 %
Goods Movement - 8.1 % 7.8 % - -
Cultural & Environmental Sensitivity
6.8 % 5.5 % 5.3 % 4.1 %11.0 % /
9.6 %
KDP2 Cookbook –TIP Project Evaluation Documentation
Guide to the TIP Project Solicitation
Documents the decisions made by the TIP Prioritization Taskforce
Outlines how ARC will technically evaluate projects, and includes a list of measures and metrics
Meant to be a companion document to the TIP application
http://www.atlantaregional.org/tipsolicitation
KDP3 – Final Factors
You can’t put a number on EVERYTHING
Preserves local decision-making, in line with SHRP C02
Another lens to account for geographic equity, social equity, jurisdictional preference
Universal TIP Project Call
Key Decision Point (KDP) 1Policy Filters
KDP 2Project Evaluation
Prioritized Lists of Regional Transportation Projects
CMAQ TAP STBGP
Project Evaluation Flowchart
KDP 3Final Factors
Outcomes
Transit
Transit 1
Transit 2
Transit 3
Trail
Trail 1
Trail 2
Trail 3
Roadway
Roadway 1
Roadway 2
Roadway 3
Technical analysis of the performance of all submitted projects
Projects are compared against similar projects to produce lists of the best projects by type
Used to help inform decision-making, not supplant it
Outcomes
KDP2 Project Tiers
Tier 1High Performance
High Cost-Effectiveness
Tier 3High Performance
Low Cost-Effectiveness
Tier 2Low Performance
High Cost-Effectiveness
Median Performance Score
Tier 4Low Performance
Low Cost-EffectivenessM
ed
ian
Co
st-
Effe
ctiv
en
ess
Sco
re
KDP2 Project Tiers
Performance ScoreCo
st E
ffec
tive
nes
s Sc
ore
Draft 2017 Project Evaluation Sneak Peak
The following results are very preliminary and reflect scores for 137 projects from ARC’s 2017 project solicitation
Only reflect completion of technical evaluation and a very preliminary staff first pass at recommendations
These will likely change as we move through KDP3 process and meet with jurisdictions
Draft 2017 Project Evaluation Sneak Peak
151%
26%
330%
413%
51% of the funding is allocated towards projects in the top tier of performance
2nd biggest piece of the pie is high performance low cost-effectiveness tier
Small slivers for tiers 2 and 4
Draft Funding Allocated by KDP2 Tier
Draft 2017 Project Evaluation Sneak Peak
Asked sponsors their priority for each project (ranked 1-5)
49% of funding is going towards top priority projects from local governments
Draft Funding Allocated by Local Priority
149%
223%
37%
49%
512%
Draft 2017 Project Evaluation Sneak Peak
The bigger the percent the better the draft projects perform compared to the entire sample for each criteria
KDP3 process should be careful about considering projects that impact: Reliability Safety Culture/Environment Land Use Compatibility
Percent Change in Scores between Solicitation Full Sample and Draft List Sample
% Change
Criteria Mean Score
Mobility 14%Reliability 6%Connectivity 39%Multi Modalism 34%Asset Management 11%Safety 5%Air Quality / Climate 25%Culture/Environment -3%Social Equity 12%Land Use Compat. 12%Good Movement 30%Employment Access 20%Final Score 13%
Draft 2017 Project Evaluation Sneak Peak
Roadway projects = 51% Transit projects = 22% Active mode projects = 27%
Asset Management = 13% System Expansion = 87%
Draft Funding Allocated by Project Type
Bike3%
Ped5%
R.Asset7%
R.Exp32%
R.TSM&O12%T.Asset
6%
T.Exp16%
Trail19%
Make sure you have enough subject matter experts to establish logical, measurable, outcome-driven measures…too many qualitative or boolean measures cloud results
Plan for this process to take a very long time and then even longer to implement the first time through
Make sure you bring your policymakers and stakeholders along for the ride…you want everyone to “own” the process in some way to ensure buy-in
Preserve a channel for local decision-making … even if the project scores come out bad, there are still reasons to build some projects
Make sure all your metrics are backed by reliable/easy-to-access data sources
Setting weights is hard. Prepare for that.
Lessons Learned
Mobility vs. Safety
Maintenance vs. Expansion
Highly congested now vs. congested in the future
Cost effectiveness vs. high performance
Difficult Trade-Offs
Contacts
Questions/Comments:
David D’Onofrio – [email protected]
Kofi Wakhisi – [email protected]
Kyung-Hwa Kim – [email protected]
