Capstone Report on Project Delivery Audits:

Highlighting Common Themes and Challenges

Report 21-06

December 29, 2021

Marlene Hartinger, Chief Auditor Meredith Frisius, Senior Internal Auditor

355 Capitol Street NE MS-31, Salem, Oregon 97301 (503) 986-4177

December 29, 2021 ODOT Executive Strategy Team Members:

Kris Strickler, ODOT Director Travis Brouwer, Assistant Director – Revenue, Finance & Compliance Cooper Brown, Assistant Director – Operations Lindsay Baker, Assistant Director – Government & External Relations Dear Executive Strategy Team Members: Since 2018 when project delivery became a priority area for audit, ODOT Audit Services has completed seven related internal audits. This capstone report is a summation of themes identified in this body of work. The report provides information for decision makers across ODOT regarding challenges impacting more than one program area. When the project delivery audits were evaluated collectively, we found three common themes:

Expanding and clarifying guidance to promote oversight and accountability

Aligning practices with goals and improving documentation to promote transparency

Improving communication to enhance continuous improvement efforts Ensuring guidance, practices, and department goals are in alignment provides ODOT the opportunity to promote transparency and accountability across the agency and with stakeholders. During the course of our audits, management was responsive to the improvement opportunities identified and took actions to implement. Incorporated throughout the report and appendix are management responses that highlight these improvement efforts. While all of the audits discussed in this report were conducted in accordance with generally accepted government auditing standards, the capstone report itself is a summation and not an audit. The report did go through our internal quality control process and was shared with management for their review and response. Sincerely,

Marlene V. Hartinger, MBA, CPA, CIA Chief Auditor

Oregon Department of Transportation

AUDIT SERVICES

i

Contents Background ..................................................................................................................................... 1

Summary ......................................................................................................................................... 2

Expanding and Clarifying Guidance to Promote Oversight and Accountability ......................... 2

Aligning Practices with Goals and Improving Documentation to Promote Transparency ......... 4

Improving Communication to Enhance Continuous Improvement Efforts ................................ 6

Conclusion ....................................................................................................................................... 7

Appendix ......................................................................................................................................... 9

Design Quality Control ................................................................................................................ 9

STIP Amendment Process Not Shown to Measurably Delay Projects ...................................... 10

Selected Information on Construction Contract Change Orders ............................................. 11

A Look at Right of Way Condemnation: Common Justifications and Varied Processing Times12

Bid Communications Met Federal and State Requirements, but Timeliness and Consistency Could Be Improved ................................................................................................................... 13

Improved Documentation Could Better Substantiate Highway Construction Contract Change Order Costs ............................................................................................................................... 14

Addenda Process ....................................................................................................................... 15

1

CAPSTONE REPORT ON PROJECT DELIVERY AUDITS: HIGHLIGHTING COMMON THEMES AND CHALLENGES BACKGROUND With the passage of Keep Oregon Moving (HB 2017), the

Oregon Legislature made a significant investment in transportation to help further the things that Oregonians value, such as a vibrant economy with good jobs, choices in transportation, a healthy environment, and safe communities. The Oregon Department of Transportation (ODOT) was charged to deliver programs and projects related to HB 2017 in an accountable, transparent, and efficient manner.

Project delivery became a significant priority for the department and a key area of emphasis in the 2018-2019 and 2020-2021 internal audit plans. To date, ODOT Audit Services has issued the following seven products related to project delivery:

Publication Number Title Short Hand

Management Letter 19-01 Design Quality Control/Quality Assurance

QC

Report 19-02 STIP Amendment Process Not Shown to Measurably Delay Projects

STIP

Report 19-03 Selected Information on Construction Contract Change Orders

CCO-Info

Report 19-04 A Look at Right of Way Condemnation: Common Justifications and Varied

Processing Times ROW

Management Letter 20-01 Bid Communications Met Federal and State Requirements, but Timeliness and

Consistency Could Be Improved

Bid Communications

Report 20-02 Improved Documentation Could Better Substantiate Highway Construction

Contract Change Order Costs CCO-Doc

Management Letter 21-03 Addenda Process1 N/A

See appendix for a brief description of each audit, page 13.

1 The Addenda Process audit did not identify significant improvement opportunities and was not included in the capstone discussion.

2

SUMMARY This capstone report draws attention to three themes that emerged from looking at the entire scope of work from the six audits that found improvement opportunities. We identified these themes based on published reports and accompanying management responses. This report highlights common challenges and emphasizes the need for an ODOT-wide response to ensure continued improvements in future project delivery. The three themes are:

1. Expanding and clarifying guidance to promote

oversight and accountability.

2. Aligning practices with goals and improving

documentation to promote transparency.2

3. Improving communication to enhance continuous

improvement efforts.

The following table identifies those reports that touch, either directly or indirectly, on the themes shown.

Audit Short Hand Strengthening Oversight and Accountability

Promoting Transparency

Enhance Continuous Improvement

QC

STIP

CCO-Info

ROW

Bid Communications

CCO-Doc

EXPANDING AND

CLARIFYING GUIDANCE

TO PROMOTE

OVERSIGHT AND

ACCOUNTABILITY

In each of the six audits, we identified the need to improve guidance to strengthen oversight and accountability. Within this theme, three areas of focus emerged: the need for updating existing guidance, implementing periodic monitoring, and conducting competency training.

2 Transparency as defined in this report includes internal transparency within ODOT and external transparency with business partners and Oregonians.

Audits with theme:

- QC

- STIP

- CCO-Info

- ROW

- Bid Communications

- CCO-Doc

3

Updating existing guidance was present in five audits (STIP, QC, ROW, Bid Communications, & CCO-Doc). Findings from the STIP audit revealed the need to better align project schedules, project controls, and guidance. Management's response to the audit highlighted the misalignment between these critical elements. According to management, ODOT project delivery has worked with key business partners and programs to develop tools and update guidance to better align these elements. This effort includes statewide standards of practice and supporting documentation for project delivery processes. The QC audit found that while guidance existed, each region took a different approach to quality control, and no region was completely compliant with published guidelines. The management response highlighted efforts underway to align all the region's quality control efforts by creating a new Project Delivery Quality Management Program. The ROW, Bid Communications, and CCO-Doc audits all had similar findings and management responses as the QC audit. These audits all found that existing documentation had insufficient guidance that may have contributed to oversight and accountability concerns. Management responses highlighted existing gaps in guidance and planned actions to correct these concerns. A recent management update stated that corrective actions included creating new programs, updating guidance, refining processes, and establishing new controls. The implementation of periodic monitoring was discussed in three audit reports and management responses (STIP, QC, & CCO-Doc). Management responses in both the STIP and QC audits referred to the implementation of the Project Development Improvement Initiative. According to a recent management update, the initiative has established new project controls and phase gates within ODOT's project development and design phases. Management responses underscored the phase gates as an opportunity for ODOT to establish standards for project delivery and provide periodic monitoring of projects, thus enhancing accountability and oversight. The current monitoring practices used to detect unbalanced bidding were identified as an area that needs improvement in the CCO-Doc audit. The initial management response highlighted

4

ALIGNING PRACTICES

WITH GOALS AND

IMPROVING

DOCUMENTATION TO

PROMOTE

TRANSPARENCY

management’s intent to conduct specification and process work related to unbalanced bid identification to improve ODOT's ability to identify and monitor for unbalanced bidding. According to a recent management update, management has drafted a white paper on the topic and work is underway with ODOT’s industry partners to implement an unbalanced bidding policy. Management aims to have the new policy in place in early 2022. This improvement will examine bids at bid opening and through the project's life. Management’s goals for the new policy are to ensure equitable and fair competition, as well as proper use of public funds. Initial management responses also raised the theme of competency training for three of the audits (CCO-Info, Bid Communications, & CCO-Doc). The management response for the CCO-Info highlighted core competency training events as an opportunity to provide needed training and regular reinforcement of CCO documentation expectations. Both the Bid Communications and CCO-Doc audits identified the quarterly and annual resident engineers meetings as opportunities to provide training. In addition, these meetings allow knowledgeable staff the opportunity to share experiences and solutions with new staff. Expanding, updating, and clarifying guidance throughout ODOT would help strengthen oversight and accountability. These improvements may require the department to consider periodic training to promote compliance. A recent management update highlighted efforts to develop curriculum and virtual training to strengthen competency for both ODOT and consultant staff. During the audit processes, a lack of consistent documentation across the five regions and headquarters was a common issue identified in five of the six reports. These audits found that the inconsistencies created challenges for the department internally and for those working with ODOT externally. While the types of inconsistencies varied across the audits, each audit identified the need to align guidance and practices to promote ODOT's goal of

Audits with theme:

- QC

- CCO-Info

- ROW

- Bid Communications

- CCO-Doc

5

transparency. The following examples highlight the misalignment of guidance, practices, and ODOT's transparency goals. During the QC audit, we found that none of the five regions completely followed the documentation requirements laid out by the Region Tech Center quality control plan. The absence of complete compliance with department guidance resulted in a lack of all required documentation and practices that didn't align with ODOT's stated transparency goals. Management expects the development and establishment of statewide quality plans will create standards of practice, processes, and documentation across regions. The Bid Communications and CCO-Doc audits found that while guidance existed, there were opportunities to strengthen practices by aligning guidance with desired goals. For example, the Bid Communications audit addressed the need to incorporate guidance into existing standards related to planholder response timeliness. The absence of such guidance has resulted in varied response times that do not align with ODOT's transparency goals. Management recognized the opportunity to update existing guidance for CCOs to promote transparency. Similarly, the CCO-Doc audit highlighted the need to align CCO documentation with ODOT’s desired level of transparency and accountability. CCO-Doc also identified inconsistent actions related to the inclusion of contractor-provided supporting documentation in CCO files. The CCO-Doc management response acknowledged the opportunity for improvement. A recent management update noted the implementation of guidance and policy changes as further outlined in the appendix of this report. The audit conducted on the Right of Way (ROW) condemnation process revealed that the desire to preserve project timelines drove the condemnation process and resulted in the misalignment between published guidance and agency practice. While the management response pointed to upstream planning issues contributing to this misalignment, the audit recommended aligning the process to actual performance metrics. In follow-up, management shared that ROW Headquarters has been working with the regions to ensure project schedules are appropriately planned for the ROW process and, if necessary, condemnation. Working together to understand the time and costs associated with these steps may lead regions to develop more accurate project schedules and reduce the need for condemnation.

6

IMPROVING

COMMUNICATION TO

ENHANCE

CONTINUOUS

IMPROVEMENT

EFFORTS

ODOT has an opportunity to evaluate if practices align with stated transparency goals. By aligning practices and improving documentation, the inconsistencies highlighted in these audits could be reduced while improving transparency. Updating guidance to ensure sufficient and consistent documentation would improve transparency across ODOT. Four of the six audits cited opportunities for communication improvements. These opportunities can be broken down into external and internal stakeholder communication. The STIP and Bid Communications audits found that ODOT could improve external stakeholder communication. The STIP audit recommended that ODOT work with Metropolitan Planning Organizations (MPOs), the Federal Highway Administration, and the Oregon Transportation Commission to identify opportunities to streamline the STIP amendment process. Management suggested forming a stakeholder collaboration group that will focus on standardizing the triggers of STIP amendments and transferring the public requirements to the MPOs to eliminate duplicative work. The changes are expected to reduce the time associated with STIP amendments and ensure ODOT and the MPOs are aligned in the process. A recent management update noted an effort is already underway to work with the MPOs to streamline processes while maintaining transparency. The Bid Communications audit revealed inconsistencies in how ODOT responds to planholder questions. Stakeholders reported delays in response times. This audit found that response times varied from same-day responses to those taking 23 calendar days. The contractors we interviewed also noted the turnover of experienced ODOT staff as a significant contributor to communication challenges. The audit recommended that the department make an effort to engage with stakeholders to evaluate best practices related to planholder communications, including how best to publish planholder questions and ODOT answers. Improving external communication efforts is likely to improve relationships with key stakeholders. According to a

7

recent management update, all the recommendations in the audit have been implemented. For details regarding actions taken see the appendix. The CCO-Info audit and the ROW audit both underscored areas for internal communication improvements. Management identified enhanced communication and training as steps that would likely improve the conditions noted in the CCO-Info audit. As a recent management update stated, quarterly project management gatherings and monthly construction peer group meetings presented an opportunity to address the communication and competency issues raised in the audit. The ROW audit identified areas for improved communication between the five regional ROW offices and ROW Headquarters. The audit recommended ROW Headquarters work with regions to better align the ROW and condemnation processes with the actual performance for payment files. The audit also recommended clearer communication related to project schedules to ensure sufficient time for ROW negotiations before files are moved forward to condemnation. The management response specifically raised the need for better communication related to upstream delays and project timelines. Focused efforts to improve internal and external stakeholder communication efforts are likely to alleviate the issues highlighted by these audits. By engaging with partners inside and outside the department, ODOT can work to build partnerships that enhance continuous improvement efforts.

CONCLUSION By examining project delivery audits and the corresponding management responses, common themes emerged that impact more than one program area within the department. Ensuring guidance, practices, and department goals are in alignment provides ODOT the opportunity to promote transparency throughout the agency and with stakeholders. ODOT can strengthen oversight and accountability through updating guidance and providing additional training. As highlighted by this report, improved communications with internal and external stakeholders has the potential to improve several aspects of ODOT project delivery. Overall, taking and applying the findings from the audited program areas allows ODOT to leverage the lessons learned and take actions to improve as one ODOT.

8

According to management, the audit reports helped ODOT project delivery and key business partners prioritize work efforts to address and implement actions that ensure transparency, strengthen oversight and accountability, and improve communications. Many of the recommendations in the audit reports have been implemented while others are continuing to be developed. Please see the appendix for updates to the actions taken for each audit.

9

APPENDIX Audit Services Reports on Project Delivery Management Letter 19-01 Publication Date: March 2019 Scope: We reviewed five projects from each of the five regions to determine if design quality control is being documented in accordance with region tech center quality control plans and the bridge design manual. Finding(s): Results showed that quality control documentation compliance varied by region.

Four of the five regions have some documentation supporting the completion of QC in

alignment with the region QC plan. The fifth region did not have documentation that aligned

with its QC plan but did have some alternative documentation supporting completion of QC. All

projects reviewed that included bridge design had documentation supporting completion of QC

in alignment with the bridge manual.

Management Response upon Publication: Management agreed with each of the

recommendations and provided details related to carrying out these recommendations.

Management Update October 2021:

The Project Delivery QA/QC program is working with the ETSB (Engineering and Technical Services Branch) to develop statewide standards of practice and implement quality assurance reviews to build statewide consistency in Project Delivery quality management.

Statewide Project Delivery Branch (SPDB) established a statewide Project Delivery

Quality Management program and is drafting a plan to provide an overview of the

Project Delivery QA/QC Program and set expectations regarding standards of practice,

roles and responsibilities, and quality assurance reviews.

Technical discipline and project management statewide quality plans providing

discipline specific quality guidance are in place, with the remaining three (Roadway,

Traffic, and Project Management) expected to be in place by the end of the year

(2021). The region quality plans are being updated for statewide consistency.

SPDB is developing guidance for consultant quality plans and how to review consultant

deliverables.

Efforts aim at having the Statewide Project Delivery Quality Management plan, technical

discipline quality plans, updated region quality plans, and consultant guidance in place

by the end of 2021. These standards of practice will form a solid foundation for aligning

statewide quality practices and a basis for establishing regular quality assurance reviews

to ensure that region QC practices align with the quality plans and provide insight for

continuous improvement.

DESIGN QUALITY CONTROL

10

Report 19-02 Publication Date: June 2019 Scope: We reviewed projects that staff reported as having been delayed by the STIP

amendment process and a subset of amendments from 49 projects.

Finding(s): Results showed that multiple factors contributed to delays, but that the STIP

amendment process itself did not cause significant impacts on project delivery schedules. The

report details the following four factors as key contributors to STIP amendments: the

Americans with Disabilities Act (ADA) settlement, Right of Way (ROW) time requirements,

inadequate information at project scoping, and the impact of House Bill 2017. This report

identified three additional factors which warrant further study related to their impact on

project delivery: the process for building the STIP, program decision making related to STIP

projects, and project controls for scheduling and funding within the project development stage.

Management Response upon Publication: Management agreed the three additional factors

identified by our audit warrant further study and management provided details related to

carrying out this review.

Management Update October 2021:

The Oregon Transportation Commission (OTC) increased thresholds for STIP amendment

actions needing commission approval and delegated approval authorities to the ODOT

director and division administrators on April 16, 2020.

Project Delivery has implemented formal change management practices and established

a tracking database to improve transparency and to identify themes that emerge from

changes in project development.

To improve transparency, bi-monthly reports are now shared with the OTC showing the

number of full STIP amendments.

STIP AMENDMENT PROCESS NOT SHOWN TO MEASURABLY DELAY PROJECTS

11

Report 19-03 Publication Date: July 2019 Scope: This report represents the first of two audits on Contract Change Orders (CCO). This

audit focused on the significance of change orders in terms of time and costs, as well as

timeliness of change order payments.

Finding(s): Results showed that for the 80 contracts reviewed, change orders when viewed

collectively increased project completion time by almost 39 percent over the original contract

dates. The report linked increased project delivery time to projects that experienced one or

more of the following three items: a change in ODOT project manager, required increased

project complexity, or a required external consultant for design and/or construction. In terms of

cost, payments for the 80 projects as a whole fell within the bounds set by the original

construction authorizations both for total project costs and contractor payments.

Management Response upon Publication: While the audit report did not provide specific

recommendations to be implemented, management noted that there were improvement

opportunities that may be addressed through additional internal training and guidance.

Management Update October 2021:

Monthly Construction Leadership group meetings and quarterly resident engineer

meetings allow for regular discussion of specific topics and experiences. This

information from these meetings are taken back to the Resident Engineer (RE)

construction crews to promote consistent contract administration among the RE offices

across the state.

Routine guidance conversations between the RE offices and the ODOT Construction

Section Contract Administration Unit.

Routine core competency training events such as the FHWA Core Curriculum classes.

The implementation of the Doc-Express document management system which allows RE

crews and contractors to access project records through a secure website, providing

greater transparency of CCO status in order to promote improved communication and

timely processing.

SELECTED INFORMATION ON CONSTRUCTION CONTRACT CHANGE ORDERS 3, July 23, 2019

12

Report 19-04 Publication Date: December 2019 Scope: We reviewed the Right of Way (ROW) condemnation process to determine if common

justifications for the use of condemnation exist and to identify the factors that determine the

amount of time needed to acquire possession of property through condemnation. This report

analyzed property acquired during calendar years 2015-2018 that had a recommendation for

condemnation.

Finding(s): Results showed that the project timeline drove the condemnation process. ROW

files were recommended for condemnation for two reasons: to ensure possession within the

project timeline and to allow ODOT to obtain possession when property owners rejected the

offer. In regard to ROW timelines, this audit showed that the 17-week ROW processing

mandate allows for ongoing negotiations. However, this mandate extends beyond what is

generally needed to obtain possession through condemnation.

Management Response upon Publication: Management agreed with the findings and provided

details regarding actions that will be implemented to inform the ROW decision process and to

improve the ROW timeline.

Management Update October 2021:

The OTC delegated authorization of Right of Way to the ODOT Director which is

expected to improve timelines.

ROW Headquarters has implemented a bi-monthly process with both the regional ROW

offices and the Director’s office. This process tracks and monitors the pre-condemnation

process to ensure adequate time is allowed for voluntary negotiations and settlements

to reduce the number of formal condemnations.

To ensure the regional ROW offices have sufficient time for negotiations, the best

practice is to fully comply with the new Design Acceptance Phase (DAP) milestone phase

gate and have legal descriptions complete. The newly instituted bi-monthly process has

enabled the regional ROW offices to communicate their need for sufficient time to

complete the acquisition process with the rest of their respective project teams.

A LOOK AT RIGHT OF WAY CONDEMNATION: COMMON JUSTIFICATIONS AND VARIED PROCESSING TIMES

13

Report 20-01 Publication Date: March 2020

Scope: We reviewed ODOT’s policies and practices for communicating with planholders during

the bid advertisement period of highway and bridge construction projects.

Finding(s): Results revealed that ODOT communication practices during the highway

construction contract bid advertisement period adhere to federal and state requirements, and

were generally consistent with practices of other DOTs. While communication was in

compliance with regulations the report details opportunities for improvement.

Management Response upon Publication: Management agreed with the audit findings and

provided details regarding actions that will be implemented to improve ODOT communication

with planholders.

Management Update October 2021:

In addition to updating Chapter 6 in the ODOT Construction Manual, the Construction

Section provided training to resident engineers in December of 2020 on how to answer

bidder questions.

Construction Section completed a planholder engagement survey with stakeholders that

was specific to best practices for answering and communicating bidder questions. The

survey was completed in September 2021 and the results will be used for additional

training for resident engineers when answering bidder questions.

The ODOT Project Controls Office has developed a memorandum providing guidance to

all ODOT and consultant project development personnel and resident engineers to

ensure consistency in answering bidder questions within reasonable timeframes. The

memorandum summarizes contract references and uses the results of the planholder

engagement survey to ensure ODOT and consultant project delivery personnel have a

consistent understanding and application of best management practices.

BID COMMUNICATIONS MET FEDERAL AND STATE REQUIREMENTS, BUT TIMELINESS AND CONSISTENCY

COULD BE IMPROVED

14

Report 20-02 Publication Date: April 2020 Scope: This report represents the second of two audits focused on contract change orders

(CCO) and focuses on the resident engineers’ approach in estimating cost changes for CCOs.

Finding(s): Results revealed that ODOT could achieve greater transparency and accountability

by improving CCO supporting documentation, improving guidelines related to evaluating

contractor cost reduction proposals, and improving steps to take to assure that unbalanced bids

are not adversely affecting highway construction contracts.

Management Response upon Publication: Management agreed with the audit findings and

provided details regarding actions that will be implemented to improve consistency across

resident engineers, enhance transparency and improve bid balance.

Management Update October 2021:

Chapter 15 in the ODOT Construction Manual has been updated to clarify the

appropriate cost information that should be included within the supporting

documentation for a CCO.

The ODOT Construction Section discusses proper CCO processing as a routine training

topic during quarterly meetings with resident engineers and their construction crews.

Resident engineers and their construction crews have been instructed to utilize

resources within the ODOT Value Engineering team to assist with the review of cost

reduction proposals.

An unbalanced bid matrix has been developed along with the completion of a related

specification update, which are intended to assist with monitoring and assessing of

unbalanced bids. These tools are anticipated to be negotiated with AGC members, and a

final policy is anticipated to be completed thereafter.

IMPROVED DOCUMENTATION COULD BETTER SUBSTANTIATE HIGHWAY CONSTRUCTION CONTRACT

CHANGE ORDER COSTS

15

Management Letter 21-03 Publication Date: October 2021 Scope: The report had two objectives. The first was to determine if the process for issuing

addenda is documented and followed by staff. The second categorized changes made to

addendum for contracts with bid let dates between January 1, 2019 and June 30, 2020.

Finding(s): This was an informational report and did not include recommendations.

Management Response upon Publication: Management noted that although the report did not

contain recommendations, the audit provided great information and provides the Project

Controls Office with an occasion to assess opportunities for continuous improvement.

ADDENDA PROCESS