Connected Electric Autonomous Vehicle (CEAV) Test Plan · 2019-10-08 · The purpose of this Test Plan is to document the strategic plan for testing and the criteria for acceptance

Connected Electric Autonomous Vehicle (CEAV) Test Plan

for the Smart Columbus Demonstration Program DRAFT REPORT | October 7, 2019

Produced by City of Columbus

Notice This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange. The United States Government assumes no liability for its contents or use thereof.

The U.S. Government is not endorsing any manufacturers, products, or services cited herein and any trade name that may appear in the work has been included only because it is essential to the contents of the work.

Acknowledgment of Support This material is based upon work supported by the U.S. Department of Transportation under Agreement No. DTFH6116H00013.

Disclaimer Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the Author(s) and do not necessarily reflect the view of the U.S. Department of Transportation.

CEAV Test Plan – DRAFT Report | Smart Columbus Program | i

Acknowledgments

The Smart Columbus Program would like to thank the CEAV project team, which includes representatives from various agencies and stakeholders, for their support and valuable input.

• Central Ohio Transit Authority (COTA)

• Ohio Department of Transportation (ODOT)

• The Ohio State University (OSU)

• St. Stephen’s Community House

• EasyMile

CEAV Test Plan – DRAFT Report | Smart Columbus Program | iii

Abstract

The purpose of this Test Plan is to document the strategic plan for testing and the criteria for acceptance of the CEAV system before the automated shuttle service is deployed. There is a three-phase approach to testing, from Factory Acceptance Testing, to Preliminary Acceptance Testing, to Final Acceptance Testing.

CEAV Test Plan – DRAFT Report | Smart Columbus Program | v

Table of Contents

1.1. Overview ............................................................................................................................. 13

1.2. References ......................................................................................................................... 14

3.1. Test Components .............................................................................................................. 17

3.2. Features to be Tested ....................................................................................................... 17 3.2.1. Full Vehicle System Performance ........................................................................................... 17 3.2.2. Storage, Charging, and Maintenance Facility ....................................................................... 17 3.2.3. EasyMile’s SOP and Other Factory, Operations, Maintenance, and Administrative

Procedures ................................................................................................................................ 18 3.2.4. Smart Columbus Operating System ....................................................................................... 18

3.3. Features not to be Tested ................................................................................................ 18

4.1. Testing Approach .............................................................................................................. 21

4.2. Testing Phases .................................................................................................................. 21 4.2.1. Factory Acceptance Testing ..................................................................................................... 22 4.2.2. Preliminary Acceptance Testing .............................................................................................. 23 4.2.3. Final Acceptance Testing ......................................................................................................... 25

4.3. Schedule ............................................................................................................................. 26

4.4. Roles ................................................................................................................................... 27

4.5. Test Tools ........................................................................................................................... 27

4.6. Measures and Metrics ...................................................................................................... 28

4.7. Test Criteria ........................................................................................................................ 28 4.7.1. Level of Autonomy Expected ................................................................................................... 28 4.7.2. Item Pass/Fail ............................................................................................................................ 29 4.7.3. Testing Suspension and Resumption ..................................................................................... 29 4.7.4. Testing in Varying Weather Conditions ................................................................................... 30

Table of Contents

vi | Smart Columbus Program | CEAV Test Plan – DRAFT Report

8.1. Factory Acceptance Testing ............................................................................................ 47

8.2. Preliminary Acceptance Testing ..................................................................................... 61

8.3. Final Acceptance Testing ................................................................................................. 79

List of Tables

Table 4-1: Dates of Proposed Testing Periods ............................................................................................ 26 Table 4-2: Project Team Member Roles and Responsibilities ..................................................................... 27 Table 6-1: CEAV System Test Cases .......................................................................................................... 33 Table 7-1: Acceptance Test Scenarios – Factory Acceptance Testing ........................................................ 41 Table 7-2: Acceptance Test Scenarios – Preliminary Acceptance Testing and Final Acceptance Testing .. 42 Table 8-1: Factory Acceptance Test 1 ......................................................................................................... 47 Table 8-2: Factory Acceptance Test 2 ......................................................................................................... 47 Table 8-3: Factory Acceptance Test 3 ......................................................................................................... 48 Table 8-4: Factory Acceptance Test 4 ......................................................................................................... 48 Table 8-5: Factory Acceptance Test 5 ......................................................................................................... 49 Table 8-6: Factory Acceptance Test 6 ......................................................................................................... 49 Table 8-7: Factory Acceptance Test 7 ......................................................................................................... 50 Table 8-8: Factory Acceptance Test 8 ......................................................................................................... 50 Table 8-9: Factory Acceptance Test 9 ......................................................................................................... 51 Table 8-10: Factory Acceptance Test 10 ..................................................................................................... 51 Table 8-11: Factory Acceptance Test 11 ...................................................................................................... 52 Table 8-12: Factory Acceptance Test 12 ..................................................................................................... 52 Table 8-13: Factory Acceptance Test 13 ..................................................................................................... 52 Table 8-14: Factory Acceptance Test 14 ..................................................................................................... 53

Table of Contents

CEAV Test Plan – DRAFT Report | Smart Columbus Program | vii

Table 8-15: Factory Acceptance Test 15 ..................................................................................................... 54 Table 8-16: Factory Acceptance Test 16 ..................................................................................................... 54 Table 8-17: Factory Acceptance Test 17 ..................................................................................................... 55 Table 8-18: Factory Acceptance Test 18 ..................................................................................................... 55 Table 8-19: Factory Acceptance Test 19 ..................................................................................................... 56 Table 8-20: Factory Acceptance Test 20 ..................................................................................................... 56 Table 8-21: Factory Acceptance Test 21 ..................................................................................................... 57 Table 8-22: Factory Acceptance Test 22 ..................................................................................................... 57 Table 8-23: Factory Acceptance Test 23 ..................................................................................................... 58 Table 8-24: Factory Acceptance Test 24 ..................................................................................................... 58 Table 8-25: Factory Acceptance Test 25 ..................................................................................................... 59 Table 8-26: Factory Acceptance Test 26 ..................................................................................................... 59 Table 8-27: Factory Acceptance Test 27 ..................................................................................................... 59 Table 8-28: Factory Acceptance Test 28 ..................................................................................................... 60 Table 8-29: Factory Acceptance Test 29 ..................................................................................................... 60 Table 8-30: Preliminary Acceptance Test 1 ................................................................................................. 61 Table 8-31: Preliminary Acceptance Test 2 ................................................................................................. 61 Table 8-32: Preliminary Acceptance Test 3 ................................................................................................. 62 Table 8-33: Preliminary Acceptance Test 4 ................................................................................................. 62 Table 8-34: Preliminary Acceptance Test 5 ................................................................................................. 63 Table 8-35: Preliminary Acceptance Test 6 ................................................................................................. 63 Table 8-36: Preliminary Acceptance Test 7 ................................................................................................. 64 Table 8-37: Preliminary Acceptance Test 8 ................................................................................................. 64 Table 8-38: Preliminary Acceptance Test 9 ................................................................................................. 65 Table 8-39: Preliminary Acceptance Test 10 ............................................................................................... 65 Table 8-40: Preliminary Acceptance Test 11 ............................................................................................... 66 Table 8-41: Preliminary Acceptance Test 12 ............................................................................................... 66 Table 8-42: Preliminary Acceptance Test 13 ............................................................................................... 66 Table 8-43: Preliminary Acceptance Test 14 ............................................................................................... 67 Table 8-44: Preliminary Acceptance Test 15 ............................................................................................... 67 Table 8-45: Preliminary Acceptance Test 16 ............................................................................................... 68 Table 8-46: Preliminary Acceptance Test 17 ............................................................................................... 69 Table 8-47: Preliminary Acceptance Test 18 ............................................................................................... 69 Table 8-48: Preliminary Acceptance Test 19 ............................................................................................... 70 Table 8-49: Preliminary Acceptance Test 20 ............................................................................................... 70 Table 8-50: Preliminary Acceptance Test 21 ............................................................................................... 71 Table 8-51: Preliminary Acceptance Test 22 ............................................................................................... 71 Table 8-52: Preliminary Acceptance Test 23 ............................................................................................... 72 Table 8-53: Preliminary Acceptance Test 24 ............................................................................................... 72 Table 8-54: Preliminary Acceptance Test 25 ............................................................................................... 73 Table 8-55: Preliminary Acceptance Test 26 ............................................................................................... 73 Table 8-56: Preliminary Acceptance Test 27 ............................................................................................... 74

Table of Contents

viii | Smart Columbus Program | CEAV Test Plan – DRAFT Report

Table 8-57: Preliminary Acceptance Test 28 ............................................................................................... 74 Table 8-58: Preliminary Acceptance Test 29 ............................................................................................... 75 Table 8-59: Preliminary Acceptance Test 30 ............................................................................................... 75 Table 8-60: Preliminary Acceptance Test 31 ............................................................................................... 76 Table 8-61: Preliminary Acceptance Test 32 ............................................................................................... 76 Table 8-62: Preliminary Acceptance Test 33 ............................................................................................... 76 Table 8-63: Preliminary Acceptance Test 34 ............................................................................................... 77 Table 8-64: Preliminary Acceptance Test 35 ............................................................................................... 77 Table 8-65: Preliminary Acceptance Test 36 ............................................................................................... 78 Table 8-66: Preliminary Acceptance Test 37 ............................................................................................... 78 Table 8-67: Preliminary Acceptance Test 38 ............................................................................................... 78 Table 8-68: Preliminary Acceptance Test 39 ............................................................................................... 79 Table 8-69: Final Acceptance Test 1 ........................................................................................................... 79 Table 8-70: Final Acceptance Test 2 ........................................................................................................... 80 Table 8-71: Final Acceptance Test 3 ........................................................................................................... 80 Table 8-72: Final Acceptance Test 4 ........................................................................................................... 81 Table 8-73: Final Acceptance Test 5 ........................................................................................................... 81 Table 8-74: Final Acceptance Test 6 ........................................................................................................... 82 Table 8-75: Final Acceptance Test 7 ........................................................................................................... 82 Table 8-76: Final Acceptance Test 8 ........................................................................................................... 82 Table 8-77: Final Acceptance Test 9 ........................................................................................................... 83 Table 8-78: Final Acceptance Test 10 ......................................................................................................... 83 Table 8-79: Final Acceptance Test 11 .......................................................................................................... 84 Table 8-80: Final Acceptance Test 12 ......................................................................................................... 84 Table 8-81: Final Acceptance Test 13 ......................................................................................................... 85 Table 8-82: Final Acceptance Test 14 ......................................................................................................... 85 Table 8-83: Final Acceptance Test 15 ......................................................................................................... 86 Table 8-84: Final Acceptance Test 16 ......................................................................................................... 86 Table 8-85: Final Acceptance Test 17 ......................................................................................................... 87 Table 8-86: Final Acceptance Test 18 ......................................................................................................... 87 Table 8-87: Final Acceptance Test 19 ......................................................................................................... 88 Table 8-88: Final Acceptance Test 20 ......................................................................................................... 88 Table 8-89: Final Acceptance Test 21 ......................................................................................................... 89 Table 8-90: Final Acceptance Test 22 ......................................................................................................... 89 Table 8-91: Final Acceptance Test 23 ......................................................................................................... 90 Table 8-92: Final Acceptance Test 24 ......................................................................................................... 90 Table 8-93: Final Acceptance Test 25 ......................................................................................................... 91 Table 8-94: Final Acceptance Test 26 ......................................................................................................... 91 Table 8-95: Final Acceptance Test 27 ......................................................................................................... 92 Table 8-96: Final Acceptance Test 28 ......................................................................................................... 92 Table 8-97: Final Acceptance Test 29 ......................................................................................................... 93 Table 8-98: Final Acceptance Test 30 ......................................................................................................... 93

Table of Contents

CEAV Test Plan – DRAFT Report | Smart Columbus Program | ix

Table 8-99: Final Acceptance Test 31 ......................................................................................................... 94 Table 8-100: Final Acceptance Test 32 ....................................................................................................... 94 Table 8-101: Final Acceptance Test 33 ....................................................................................................... 95 Table 8-102: Final Acceptance Test 34 ....................................................................................................... 95 Table 8-103: Final Acceptance Test 35 ....................................................................................................... 96 Table 8-104: Final Acceptance Test 36 ....................................................................................................... 96 Table 8-105: Final Acceptance Test 37 ....................................................................................................... 96 Table 8-106: Final Acceptance Test 38 ....................................................................................................... 97 Table 8-107: Final Acceptance Test 39 ....................................................................................................... 97 Table 8-108: Final Acceptance Test 40 ....................................................................................................... 98 Table 8-109: Final Acceptance Test 41 ....................................................................................................... 98 Table 8-110: Final Acceptance Test 42 ........................................................................................................ 98 Table 8-111: Final Acceptance Test 43 ........................................................................................................ 99 Table 8-112: Final Acceptance Test 44 ........................................................................................................ 99 Table 8-113: Final Acceptance Test 45 ...................................................................................................... 100 Table 8-114: Final Acceptance Test 46 ...................................................................................................... 100 Table 8-115: Final Acceptance Test 47 ...................................................................................................... 100 Table 8-116: Numbering Convention Definitions ....................................................................................... 103 Table 8-117: Acronym List ......................................................................................................................... 105 Table 8-118: Glossary ............................................................................................................................... 107 Table 8-119: Requirements ....................................................................................................................... 109

List of Figures

Figure 4-1: Factory Acceptance Testing Test Track .................................................................................... 23 Figure 4-2: Preliminary Acceptance Testing Location ................................................................................. 24 Figure 4-3: Proposed Operations and Final Acceptance Testing Route ..................................................... 25 Figure 8-1: Numbering Convention ........................................................................................................... 103

CEAV Test Plan – DRAFT Report | Smart Columbus Program | 11

Executive Summary

PROJECT BACKGROUND

In June 2016, the City of Columbus, Ohio won the United States Department of Transportation (USDOT) Smart City Challenge. Columbus intends to define what it means to be a “Smart City” and serve as a model for other cities wishing to fully integrate innovative technologies, such as automated and connected vehicles into the transportation network. The Connected Electric Autonomous Vehicle (CEAV) project is one of eight projects in the Smart Columbus program. Smart Columbus aims to deploy and evaluate automated shuttles against a series of use cases in a series of pilots commissioned by the City of Columbus, the Ohio State University (OSU), the Columbus Partnership, the Ohio Department of Transportation (ODOT) through its DriveOhio initiative and coordinated with the Central Ohio Transit Authority (COTA).

The proposed technology solution involves vehicles that are Level 4 automated, as defined in SAEJ30161, electric, and connected, serving the public on short trips where other modes are not presently available or convenient. Operations of the fleet are expected to be similar to that of a traditional transit service, with predetermined routes and signed stops along the routes for passengers to board and alight. The success of this project will be looked at as a guide for the potential deployment of future CEAV routes in other parts of Columbus and elsewhere by verifying their ability to perform as intended and providing feasibility for their use in similar environments.

PROJECT OBJECTIVES

This CEAV shuttle deployment will benefit the region by demonstrating the potential of this emerging technology to local stakeholders and the public, allowing for an educational experience while also inspiring quicker adoption of future innovations. In relation to this Test Plan, results of this project will be used to inform the following overall project goals:

• Better connect the community to services through first-mile/last-mile/only-mile connections by providing a convenient and reliable transit option;

• Validate and ensure that emerging transportation technology solutions provide equitable and accessible transportation;

• Develop a set of AV operational testing and evaluation guidelines to benchmark AVs; • Summarize lessons learned to help identify needs, understand how to garner user acceptance of

systems, and study which interfaces work best.

Performance will be evaluated through the collection of trip-generated data, information on vehicle performance, and user surveys and evaluation.

1 SAE International, J3016_201806: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, revised June 2018, https://www.sae.org/standards/content/j3016_201806/.

0. Executive Summary

12 | Smart Columbus Program | CEAV Test Plan – DRAFT Report

SYSTEM CONCEPT

The Smart Columbus team is interested in deploying CEAV technology to evaluate the ability of this technology and associated vehicles to enhance the mobility of residents and visitors, to operate on public roadways in Ohio, and to satisfy the specific operating purposes for which each service is intended. Further, the team is interested in better understanding the infrastructure required to implement and support the operation of this technology, the approach to public adoption, the types and value of data produced, the associated cost, and the benefits derived from the use of CEAVs. To minimize risk and to provide the best opportunity for understanding these factors, the Smart Columbus team decided to procure turnkey solutions from vendors responsible for deploying, operating, and maintaining the services and vehicles. The Smart Columbus team will need to verify that the chosen vendors are capable of providing what they have advertised, and that they have responsible policies and procedures in place to ensure the safety and reliability of the technologies they are providing.

This Test Plan focuses specifically on the second deployment of the CEAV project, which is located in the Linden neighborhood. The chosen route travels between St. Stephen’s Community House and Linden Transit Center. The goal of this route is to connect the community center with the CMAX high-frequency bus rapid transit (BRT) line as well as the neighboring community. The CEAVs will be operated on open, public roads on a route that was designed with stakeholder input to provide first-mile/last-mile (FMLM) transit access to community services, including the Douglas Community Recreation Center and Rosewind Resident Council.

The Smart Columbus team has contracted EasyMile to provide the outlined services, and there are three phases of testing required: Factory Acceptance Testing, Preliminary Acceptance Testing, and Final Acceptance Testing.


Introduction

This Test Plan has been prepared for the verification of CEAV project elements in order to commission the system for operation. Verification evaluates the functional deployment against its intended function and implementation of the system requirements. The Test Plan will accomplish this by establishing a set of test cases intended to evaluate the degree to which each amenity and its functions conform to the requirements set forth in the procurement documentation. The Test Plan will also develop test scenarios to verify the status of technical progress on the project, substantiate the achievement of technical performance, and certify operational readiness to move forward with validation.

The Test Plan will establish a common framework for testing. This includes describing various testing methods and setting expectations of which method(s) should be used and when. The document provides information on risks and contingencies (Chapter 2), clarifies what system components and features will be tested and what will not be tested (Chapter 3), identifies the testing approach (Chapter 4), outlines Test Deliverables (Chapter 5), and provides testers with a set of Test Cases (Chapter 6) and specific Test Scenarios (Chapter 7) and Test Procedures (Chapter 8) intended to evaluate the degree to which system functionality conforms to the procurement requirements and user needs.

Due to the unique characteristics of this project, multiple phases of testing will be conducted to ensure holistic, successful outcomes. The Test Plan will provide guidance on amenity verification, employing one or more of the following methods as appropriate: (a) testing information systems characterized by data and communications, (b) testing usability in terms of transportation safety and mobility, (c) testing operational readiness with a focus on infrastructure, and (d) testing policy and compliance across all elements of the system. Additional information surrounding these testing approaches will be addressed in the approach section of this document and will be further defined in the Test Plan report matrix.

Upon completion of the three phases of testing activities, a summary of the project’s test results (along with associated defects report and other observations) will be prepared as a separate Test Report.

1.1. OVERVIEW The CEAV Test Plan incorporates information on EasyMile’s product system functionality, any enhancements implemented specifically for the Smart Columbus CEAV project, and the individual functions the system must perform to meet the intended system functionality. The CEAV project involves the procurement, validation testing, and assessment of an automated shuttle service, and not the development thereof. This project therefore follows an agile systems engineering method. Building off preliminary research and prior experience, the Smart Columbus team began by creating an Operational Concept (OpCon) to describe the development of the project and to summarize a high-level view of the system to be deployed. This document was not intended to fulfill the purpose of a more traditional Concept of Operations (ConOps) or other standard systems engineering documentation, because the technology was not being designed from the ground up and rather being procured as a turnkey service. The OpCon was instead intended to provide a quick reference for project stakeholders to ensure a consistent understanding of project needs, process framework, and other system attributes and to inform procurement documents. This was then supported by the development of requirements to be met by a vendor, outlined in the Request for

Chapter 1. Introduction


Proposals (RFP) and further refined during the procurement process, including by cross-referencing with the statewide requirements for human driver licenses.2 These requirements are presented in Appendix D.

The Test Plan outlines the three phases of testing first introduced in the OpCon and included in the RFP: Factory Acceptance Testing, Preliminary Acceptance Testing, and Final Acceptance Testing. These individual phases of the process each build upon information and data from prior phases, and therefore constraints on and functions to be demonstrated during each phase will depend on the results of each of the preceding phases. This Test Plan will ensure that the goals, objectives, and specifications that were developed and agreed upon throughout the project are all being tested and verified.

1.2. REFERENCES The following documents provide additional background on the CEAV project and should be referred to for details on user needs, requirements, and contractual agreements that guided the development of this Test Plan.

City of Columbus Solicitation Number: RFQ011174, Request for Proposals for Automated Vehicle Shuttle Service, Smart Columbus, Capital Improvements Project No: 530163 – 100020, January 17, 2019.

Connected Electric Autonomous Vehicle (CEAV) Operational Concept for the Smart Columbus Demonstration Program, Final Report, March 28, 2019.

AV Shuttle Deployment Lessons Learned for the Smart Columbus Demonstration Program, Final Report, March 29, 2019.

Professional Services Contract Between City of Columbus, Ohio And EasyMile Inc. for Smart Columbus – Connected Electronic Autonomous Vehicle Capital Improvement Project 530163-100020, June 18, 2019.

EasyMile User Guide, Generation 3 EZ10, October 2019.

This Test Plan was adopted from the IEEE 829-2008 Standard for Software and System Test Documentation.

2 Ohio Bureau of Motor Vehicles, Driving and Skills Test, https://www.bmv.ohio.gov/dl-driving-tests.aspx


Risks and Contingencies

There are several parts of the CEAV project that are not within the control of the project team but have direct impacts on the process and must be validated. The list below outlines risks and contingencies for the CEAV project:

• Integration with external systems and other Smart Columbus projects such as the Multimodal Trip Planning Application (MMTPA), Connected Vehicle Environment (CVE), or the Operating System may present risks that lead to schedule delays during testing. Testing for the MMTPA project and the CVE are both scheduled to occur at a later date than testing for the CEAV project, and the CEAV system is able to safely operate without being integrated with either of those systems. Because of this, integration testing for the two projects will be included in the MMTPA and CVE Test Plans and testing activities rather than those for the CEAV. However, there is a requirement that the CEAV provide a real-time tracking API, so the ability of the CEAV system to provide the data for this is included in a test case in this Test Plan – testing data availability, but not system integration.

• The CEAVs will come prepackaged and tested in the manufacturer’s plant, and any related equipment external to the vehicles will be installed according to the safety requirements of the CEAV quality management plan. This includes software and hardware (LiDAR, etc.), components of which will be tested to the extent possible given the complexity of the technology and proprietary concerns.

• A degraded or hacked communications network (for DSRC, for real-time location information, for data collection, etc.) could cause operational safety issues, such as miscommunication between the RSU and OBU because of radio interference issues, reduced power, capacity exceeded, occlusion, etc. The CEAV is only to ingest traffic signal information, not use or process it, and emit a Basic Safety Message for other vehicles so any network degradation will not affect the performance or safety of the vehicle.

• Resource availability issues or unfilled positions may affect the testing schedule, leading to a delay in acceptance testing. The City will leverage additional resources from the Smart Columbus consultant contract for additional support. Testers will also be required to leverage the test tools, which will provide an up-to-date snapshot for new testers to quickly get up to speed on test procedures, expectations, and progress.

• Unpredictable/unusual weather and traffic conditions could affect testing procedures and may result in schedule delays if tests need to be extended and/or rescheduled.

• Regulatory and permitting related delays specific to navigating how to legally deploy this emerging technology in an evolving policy landscape could lead to delays in testing and deployment, though time has been padded into the schedule to allow for this to a certain extent.

• Operational risks related to automated vehicle technology, including the impact of vehicle speed limits, AVs operating at low speed (slower than 25 mph) with vehicles at higher speeds (exceeding the posted speed limit), stopped operations of the vehicles, tampering with devices, mixed-traffic environment challenges, and inadequate training of operators for an encountered scenario may arise during testing and/or operations. Specific issues and mitigation strategies are detailed in the Smart Columbus Safety Management Plan. Traffic calming measures, speed enforcement, AV

Chapter 2. Risks and Contingencies


information signage, and route design will help mitigate these risks, as will CEAV operator training. EasyMile’s approach to meeting these challenges will be validated during the testing process, but there still may be unexpected operational circumstances present after service launch.


Test Items

This chapter contains an overview of the components of the system that will be tested, as well as justification for those that will not be tested.

3.1. TEST COMPONENTS The following list of components are target components planned for testing.

1. Full vehicle system performance of EZ10 shuttles a. Two EZ10-Gen3s

2. Storage, charging, and maintenance facility 3. EasyMile’s Standard Operating Procedures (SOP) and other factory, operations, maintenance, and

administrative procedures a. Many test cases and procedures refer to EasyMile’s SOP and other procedural

documentation. For these test cases and procedures, the Smart Columbus team will review the SOP or the appropriate document and ensure it is complete.

4. Smart Columbus Operating System a. To verify data is sent and received.

3.2. FEATURES TO BE TESTED The components for the system are grouped into categories, based on their characteristics and uses. In order for the system to function properly and optimally, all components of the system must function properly and must integrate with one another in a seamless and intuitive fashion. The individual functions and capabilities for each of these components are broken out and listed in the following sections. Details on specific Test Cases and a full list of features to be tested is included in Chapter 6.

3.2.1. Full Vehicle System Performance The test runs conducted during all three stages of testing will be comprehensive and will in essence test the full system as it will operate once it is approved for public deployment. For Final Acceptance Testing, this includes servicing the passengers stops (with passenger trips only being provided to members of the project team and pre-approved volunteers), operating in mixed traffic, and demonstrating procedures that support service (such as storage and charging).

Vehicle component testing will not be conducted, because the intent is to procure a turnkey system; however, to the extent possible, individual components of the vehicle system will be visually inspected for any defects and tested for functionality. This includes vehicle-based sensors and other equipment, passenger-facing systems such as onboard screens, and communications to and from the vehicles.

3.2.2. Storage, Charging, and Maintenance Facility A facility has been designated specifically to support this service, including overnight storage, charging, and some light maintenance, such as inflating tires, performing a software update and data download, cleaning the interior, cleaning external sensors, and using a jack to replace a tire. This facility will be inspected to

Chapter 3. Test Items


ensure it meets the requirements outlined in the procurement documentation and that it adequately supports the needs of the system.

EasyMile is providing a turnkey solution, so the City’s goal is to make sure it is providing what is agreed to in the contract – which is to provide vehicles that are sufficiently charged to support the service requirements. The contract did not specify features like voltage or time to charge, just what hours of operation need to be supported. Therefore, testing of the facility will be similar to an audit, where the project team will witness and sign off on a demonstration of the equipment – tasks include inspecting that there is enough space allocated for the vehicles and that when each vehicle is plugged in, it charges.

3.2.3. EasyMile’s SOP and Other Factory, Operations, Maintenance, and Administrative Procedures

During Factory Acceptance Testing, the project team will strive to ensure the policies and procedures used by vendors under contract to provide a product to support this service are satisfactory to the project team. During Preliminary and Final Acceptance Testing, similar verification of operations, maintenance, and administrative procedures will be conducted. This includes review of relevant documents as well as participation in a tabletop exercise with project stakeholders to demonstrate the step-by-step processes that would be followed for certain types of situations.

3.2.4. Smart Columbus Operating System While the design and integration of the Smart Columbus Operating System is covered by a separate Smart Columbus project, testing for the CEAV project will include an analysis of the ability of the CEAV system to collect and send data to the Operating System in real-time or near real-time or on a daily or weekly basis, depending on the type of data and the agreed-upon requirements thereof. This will also enable current vehicle location information to be made available to potential passengers in real-time or near-real time.

3.3. FEATURES NOT TO BE TESTED There are some items that will not be specifically addressed during testing. In most cases, functionality in these areas will be indirect, inferred, or assumed as a result of other testing efforts.

• Stress testing: The vehicles will be tested for operational performance but will not undergo benchmarking for stress testing.

• Network security: Individual transactions or packets will not be inspected. It is sufficient to accept the secure sockets layer (SSL) protection configured, which secures communication to the end user.

• Roadway support infrastructure: The vehicles have been designed to operate in the infrastructure as it exists, but some modifications may be made to ensure operations are safer and more user-friendly. Infrastructure such as stop signage and other stop-based infrastructure, signage to indicate the presence of low-speed vehicles along the roadway, and similar will not be tested as part of this project. However, it should be noted that infrastructure proposed as part of EasyMile’s Site Assessment Report will be inspected against City standards as is done for any infrastructure modifications in the city.

Chapter 3. Test Items

CEAV Test Plan – Draft Report | Smart Columbus Program | 19

• Lane changes: The RFP included the requirement that the vehicle be capable of changing lanes (both left and right lane changes). This has been recorded as requirement CEAV-VEH-021-v01 in the requirements summary included in Appendix D. However, the service route consists exclusively of two-lane roads. Therefore, no lane changes will be required as they are not possible to test within this infrastructure environment and they are not applicable to the deployment route.

• Roadside units: These will be tested as part of the CVE Test Plan and testing activities. The DSRC-enabled intersection is not along the route of the vehicle, and rather just nearby, with data being sent and received during deployment for demonstration purposes and not for any safety or operational uses.

• EasyMile’s onboard unit: The CVE will not be operational at the time of CEAV deployment so the OBU will be tested as part of the CVE project when it does deploy. EasyMile will be providing the OBU, but the OBU will not impact the functionality of the CEAV system (since the vehicles will not be actively using any information they send or receive), so there are no safety or operational concerns associated with this approach.

• Unit and component testing: While safety-critical pieces of equipment may be inspected, EasyMile is providing a turnkey solution, so individual pieces of equipment including sensors, brakes, and tires will not be tested individually.


Approach

This section provides a more in-depth overview of the testing approach.

4.1. TESTING APPROACH To deliver the CEAV system in an efficient way, an agile version of the systems engineering V-model approach is used. For projects using any systems engineering approach, documentation is an important aspect, as the outputs generated at a particular phase most often become inputs into the next phase, providing guidance, constraint, and governance on what to expect for the next phase. There are four different methods of testing verification used for this project: inspection, demonstration, test, and analysis. Per the INCOSE Systems Engineer Body of Knowledge3, these four methods can be defined as follows:

Inspection Technique based on visual or dimensional examination of an element; the verification relies on the human senses or uses simple methods of measurement and handling. Inspection is generally non-destructive, and typically includes the use of sight, hearing, smell, touch, and taste, simple physical manipulation, mechanical and electrical gauging, and measurement. No stimuli (tests) are necessary. The technique is used to check properties or characteristics best determined by observation (e.g. - paint color, weight, documentation, listing of code, etc.).

Demonstration Technique used to demonstrate correct operation of the submitted element against operational and observable characteristics without using physical measurements (no or minimal instrumentation or test equipment). Demonstration is sometimes called field testing. It generally consists of a set of tests selected by the supplier to show that the element response to stimuli is suitable or to show that operators can perform their assigned tasks when using the element. Observations are made and compared with predetermined/expected responses. Demonstration may be appropriate when requirements or specification are given in statistical terms (e.g. mean time to repair, average power consumption, etc.).

Test Technique performed onto the submitted element by which functional, measurable characteristics, operability, supportability, or performance capability is quantitatively verified when subjected to controlled conditions that are real or simulated. Testing often uses special test equipment or instrumentation to obtain accurate quantitative data to be analyzed.

Analysis Technique based on analytical evidence obtained without any intervention on the submitted element using mathematical or probabilistic calculation, logical reasoning (including the theory of predicates), modeling and/or simulation under defined conditions to show theoretical compliance. Mainly used where testing to realistic conditions cannot be achieved or is not cost-effective.

4.2. TESTING PHASES Testing will be conducted over three phases: Factory Acceptance Testing, Preliminary Acceptance Testing, and Final Acceptance Testing. Additional detail on these three phases is provided in this section.

3 https://www.sebokwiki.org/wiki/System_Verification

https://www.sebokwiki.org/wiki/System_Verification

Chapter 4. Approach


For each testing phase, each shuttle must complete the specified number of runs or total distance traveled without any critical failures. A critical failure is defined as:

• The shuttle’s inability to perform an essential function as agreed upon in the procurement documentation.

o Note that, as further described in Chapter 8, for some functions requiring a human operator to regularly perform a manual check (human-in-the-loop) before allowing the vehicle to proceed is acceptable, assuming the vehicle proceeds on its own once it is told the manual check is complete.

• The shuttle performing an illegal maneuver. • The shuttle taking an action that results in a crash or a crash-imminent situation.

Any incident where the CEAV is not at fault or it behaves as expected does not constitute a critical failure.

If a critical failure does occur, the incident will be further assessed for the amount of time appropriate for the severity of the incident. Once this assessment is complete, if the Smart Columbus team feels confident that the cause has been resolved and the safety and competence of the system has been restored, the testing phase will be restarted from the beginning. If the Smart Columbus team no longer holds full confidence in the system, other actions may be taken as specified in the contract.

4.2.1. Factory Acceptance Testing The intent of the testing at the factory in Toulouse, France is to verify the ability of the EZ10 shuttles to function as advertised as well as to ensure the factory itself has the proper procedures in place for quality assurance, quality control, back testing, configuration management, shakeout, and calibration. Vehicle functional verification includes both component verification (brakes work, sensors work, power train works, etc.) and full system verification (vehicles can operate on a road/test track). The EasyMile EZ10 Acceptance Test Manual can be found in Appendix E and outlines the tests conducted at the factory. The factory test track is shown in Figure 4-1. The test track pre-programmed route has three stops, and testing will be conducted in loops until 50 kilometers total have been conducted.

Factory Acceptance Testing involves a comprehensive inspection performed on equipment to verify the integrity of the equipment, that it conforms to system requirements, functionality, exposure to weather conditions, and meets other contractual specifications. In general, Factory Acceptance Testing provides proof of functionality early in the process as well as a verification of implemented testing processes by the vendor, helping to avoid costly risks due to equipment failures on delivery.

Chapter 4. Approach


Figure 4-1: Factory Acceptance Testing Test Track

4.2.2. Preliminary Acceptance Testing Immediately upon vehicle delivery, tests on the delivered vehicles similar to those done during Factory Acceptance Testing will comprise the first portion of Preliminary Acceptance Testing. These tests will be performed in a controlled test environment, likely the eastern (rear) parking lot at St. Stephen’s as shown in Figure 4-2.

Chapter 4. Approach


Figure 4-2: Preliminary Acceptance Testing Location

Once these functions have been sufficiently demonstrated on this closed course, Preliminary Acceptance Testing will graduate to being conducted on the route in Figure 4-3 under normal conditions. An operator and members of the testing team will be onboard each vehicle, but it will not be open to public passengers. Once each vehicle has completed the required test runs and is able to adequately demonstrate each operational function, it will be permitted to proceed to the full operational testing period on the service route.

Preliminary Acceptance Testing is more focused on the ability of the system to function holistically. These tests will also assess the usability of the system by way of demonstration from the perspective of potential passengers as it relates to how well the system meets transportation and safety goals. This will be accomplished through the scenarios defined in Chapter 7. For instance, testing requires asking questions such as:

• Is a user able to easily board and alight the vehicle? • Can a user connect to the Wi-Fi? • Does a user feel safe on the vehicle? • If there are challenges, how quickly can the user find answers or work around issues?

Answering these questions helps determine the usability of the system. The project team will also test the adherence to policy and compliance between and across all elements of the system either through physical inspection, testing, or analysis. Lastly, the team will ensure the series of tests demonstrate that the vehicles and the CEAV system are operationally ready. Once the CEAV system has passed Preliminary Acceptance Testing, EasyMile will be permitted to proceed to the full operational testing period. During this period, EasyMile will be permitted to operate on the route without passengers to calibrate the system and prepare for Final Acceptance Testing.

Chapter 4. Approach


4.2.3. Final Acceptance Testing Upon successful testing and demonstration of the vehicles in Factory Acceptance and Preliminary Acceptance Testing, EasyMile will be permitted to proceed to the full operational testing period to demonstrate and refine the ability to operate as desired. During this phase, the vehicles will operate per the planned route and schedule but without taking on passengers. Staff from the City, or its designees, will be permitted to ride the vehicles during this period and could simulate passenger boardings and alightings. The proposed route is shown in Figure 4-3.

Figure 4-3: Proposed Operations and Final Acceptance Testing Route

Also during this time, a tabletop exercise will be conducted with project stakeholders, such as the local transit agency, emergency responders, and transportation and traffic management officials, in which all parties will demonstrate the step-by-step processes they would undertake for certain types of situations as presented to them in the exercise. This tabletop exercise is expected to be a 3-4-hour event that is conducted near the end of the operational testing period.

Upon completion of the full operational testing period and the tabletop exercise, EasyMile and the City will prepare and conduct a Final Acceptance Test, similar to the Preliminary Acceptance Test. The testing should demonstrate the vehicles’ ability to perform everyday operations on the route, demonstrating lane adherence, turning, stopping and starting, and safe reactions to situations the vehicle happens to encounter. This includes management of the system via a cloud-based platform, service of the passenger stops (including pulling over, opening and closing doors, merging back onto the route, and servicing a passenger with the ADA accessibility ramp) and operations in changing roadway/weather conditions (such as peak and

Chapter 4. Approach


off-peak traffic). The Final Acceptance Testing will be sensitive to site-specific conditions such as negotiation with parked cars and navigation through traffic circles and parking lots.

For Final Acceptance Testing, each vehicle will perform five full test laps of the route in line with the Test Procedures in Chapter 8 without any critical failures. A Vehicles Setup Acceptance Certificate will then be signed off by the City of Columbus and EasyMile staff to confirm operational characteristics (maximum speed, operating hours, etc.), acknowledge that site setup has been completed, and provide EasyMile with permission to begin operations on the date of signature. Once Final Acceptance Testing is completed, EasyMile will then contribute to the Test Report. The Test Report will include any corrective actions necessary.

4.3. SCHEDULE Table 4-1 provides the periods of time over which each testing phase may occur. Note that the actual testing for each phase, as witnessed by Smart Columbus staff, may only take a few hours to a day. However, these extended testing periods are provided to allow EasyMile time to prepare for each test and to allow Smart Columbus staff the ability to ask questions and witness performance under varying weather and other operational conditions.

For the full operational testing period, the period of time includes the time for EasyMile to program the route into the delivered vehicles and perform preliminary operations on-route (without passengers), before culminating in Final Acceptance Testing at the end of the period. Smart Columbus staff may witness testing at any time throughout this period but will not conduct a full assessment until Final Acceptance Testing occurs. Upon completion of Final Acceptance Testing, service is expected to launch on November 19, 2019.

Table 4-1: Dates of Proposed Testing Periods

Testing Phase Testing Dates

Factory Acceptance Testing July 29, 2019 to October 11, 2019

Factory Walk-Through July 29, 2019 to August 1, 2019

Test Track Demonstrations October 7, 2019 to October 11, 2019

Simulation Testing August 1, 2019 to December 31, 2019

Preliminary Acceptance Testing November 18, 2019 to November 22, 2019

Closed Course Testing November 18, 2019 to November 19, 2019

On-Route/Open Road Testing November 20, 2019 to November 22, 2019

Full Operational Testing Period November 25, 2019 to December 6, 2019

Tabletop Exercise Week of December 9, 2019

Final Acceptance Testing December 9, 2019 to December 13, 2019

Public Operations Commence December 16, 2019

Chapter 4. Approach


Inclement Weather Testing As such weather becomes available (may be during testing period or may be after launch)

Source: City of Columbus

4.4. ROLES Anticipated tester roles and responsibilities for project team partners are outlined in Table 4-2.

Table 4-2: Project Team Member Roles and Responsibilities

Tester Roles Responsibility

System Owner City of Columbus representative overseeing the completion of all projects and testing.

Andy Wolpert, City of Columbus

Test Manager Develops testing report summaries and scheduling.

Jeff Kupko, Michael Baker

Data Consumer An authorized individual (or service) that consumes trip and travel data from the OS.

Smart Columbus Operating System Team

Test Conductor The entity that has provided the technology to be tested and will be programming and controlling the technology throughout testing and operations.

EasyMile (including the onboard operators it is responsible for providing and training)

Data Producer A system or systems that will produce usage data to be sent to the OS.

EasyMile

Stakeholders General testing role with domain knowledge in the area of transportation.

Jeff Kupko, Michael Baker

Thomas Timcho, WSP

The Ohio State University


4.5. TEST TOOLS The CEAV project will follow this Test Plan’s guidelines and EasyMile’s general procedures to manage the test cases, scenarios, procedures, testing outcomes, and maintain traceability back to user needs and procurement requirements. Testers should read each requirement description carefully to enhance their understanding of each test objective. For each test case, the matrix will also provide test steps and

Chapter 4. Approach


expected results per objective. Finally, the matrix will assign the type of role to test the case. Testers will be required to enter their name, run date, result, and comment where applicable.

Additional testing tools include cones, laptops or tablets (to record results), and any other equipment necessary to simulate the testing procedures (such as a pedestrian, scooter, parked car, smartphone with the Pivot app installed, etc.). In addition, OSU has offered its simulation tool in support of this project. This tool will be used to further assess the abilities of the automated system and to inform process and procedures prior to Final Acceptance Testing.

4.6. MEASURES AND METRICS The project team will capture the following testing metrics:

• Total number of test cases

• Number of tests runs per case

• Number and percentage of test cases passed

• Number and percentage of test cases failed

• Number and percentage of test cases deferred

• Number and percentage of defects found (relative to total cases)

• Number and percentage of high severity defects

• Number and percentage of defects accepted

• Number and percentage of defects rejected

• Number and percentage of defects deferred

• Total number of testers

• Defects, or anomalies, incongruencies, errors, or any other output inconsistent with the expected test case result

The City of Columbus will leverage these data points to determine the feasibility and operational readiness of the CEAV project to receive final acceptance test approval.

4.7. TEST CRITERIA

4.7.1. Level of Autonomy Expected EasyMile’s EZ10 is a Level 4 automated vehicle, as defined by the Society of Automotive Engineers, that can operate independently under some, but not all, circumstances and conditions. Therefore, there needs to be a delineation between times when an onboard operator is required to intervene because the vehicle is outside its operational design domain (ODD) and times when the vehicle is actually not operating as it is designed (and the onboard operator essentially has to intervene without being prompted for a safety-critical intervention). The project team has therefore decided to distinguish between instances where the vehicle asks for guidance (because it is outside its ODD) and instances where the onboard operator unexpectedly has to step in.

The first type of instance includes checkpoints along the route where the vehicle may need an operator to push a button to confirm that it is safe to proceed, such as at a relatively busy intersection (an “operator in

Chapter 4. Approach


the loop” type of safety management), as well as times when the vehicle senses it is outside its ODD, such as if there is a construction truck blocking the roadway. These types of situations are not considered critical failures.

The second type of instances are considered critical failures, and were defined in Section 4.2. This includes functional failures, such as the shuttle not being able to take a charge, illegal maneuvers, such as the shuttle not stopping at a stop sign, and crashes and crash-imminent situations, such as the onboard operator seeing an obstacle the shuttle does not and intervening to stop or otherwise control the shuttle. This will allow EasyMile to demonstrate that their shuttles, with the occasional assistance by an onboard operator, are capable of operating within the rules of the roadway.

Each vehicle needs to be capable of performing all automated functions as designed 100% of the time, and also needs to be able to know when it is outside its ODD and does not know what to do.

4.7.2. Item Pass/Fail Each test case and procedure consists of several unique properties which should be considered holistically during the testing evaluation process. Properties include but are not limited to: test identifier (ID), test objective, procedure, expected outcome, number of test runs that must be completed, and status. The results matrix will maintain the following status for each test case:

1. Planned – the test case has been defined, role identified, testers assigned, and is in the backlog ready for testing.

2. In Progress – one or more tester(s) have started testing the case.

3. Passed – a passed value indicates tests have completed the defined number of runs by various testers without error and the expected result has been achieved. It is expected that each time this test is performed, independent of who is testing, the same successful results will be achieved. There may be instances when a tester identifies a defect during the procedure, yet the test case still achieves the stated outcome. The case can still pass, but the testers must log the defect and bring it to the attention of the test leader. This can happen when there are minor bugs detected not critical to the essential functionality of the feature being tested such as a misspelling of a stop location on the onboard monitor. Similarly, this can occur when the pass criteria metric for a particular test is lower than 100%.

4. Failed – a test case is marked as failed when it does not meet or partially meets the expected outcome. Some test scenarios encompass several steps. If some steps are successful, but others are not, this would be an example of a failed test case that partially met the expected outcome, which the tester would mark as failed. For all failed test cases, one or more defects must be logged to capture the details surrounding the failure.

5. Deferred – the test case is marked as deferred when it is unable to be performed at the current state. Most often this will occur when a product is being released in increments and the functionality is not ready when it’s time to test the current release. Deferred could also mean that a requirement has changed and therefore the test is no longer needed. If a test is deferred, the tester should provide a brief reason why in the comment column. The test lead is responsible for tracking deferred cases and evaluating the most appropriate time and/or response for addressing the case.

4.7.3. Testing Suspension and Resumption There are cases when a critical, severe defect is detected that is significant enough that if it is not fixed and testing proceeds, it would need to be performed again. In these situations, it is better to suspend testing until

Chapter 4. Approach


the defect is resolved to prevent wasting the project budget and the testers’ time. The test leader should be notified immediately and will work with the vendor to correct the issue as quickly as possible. Testing will resume once the test leader has successfully confirmed the issue has been resolved.

The following situations would cause testing to be suspended:

1. One or more defects found associated with the structural integrity of the vehicles.

2. Failure of communications network rendering vehicle isolated from systems required for normal operation.

3. Any situation that could potentially lead to bodily injury or significant damage to property.

4.7.4. Testing in Varying Weather Conditions The RFP required that the selected vendor be capable of operating in rain, fog, and light snow conditions (snow fall and snow pack on the side of the road) not deemed a weather emergency. However, given the seasonal variation of weather conditions in Central Ohio, it is likely that this entire set of weather conditions will not be available during the proposed testing dates. Therefore, the vehicles will be permitted to proceed to public operations even if the team is not able to test under all of these conditions during the testing period. On the first day that these conditions become available, testing will recommence, and the vehicles will be required to pass at that point before they are permitted to operate under such conditions. In addition, guidance from other projects (in particular the Minnesota Automated Shuttle Bus Pilot Project), other documentation from EasyMile, and simulation by OSU will be utilized to set expectations for vehicle performance under varying weather conditions.


Test Deliverables

The USDOT and City of Columbus Cooperative Agreement identifies one deliverable for the CEAV project, specifically the delivery of this Test Plan document. However, during testing there will be additional artifacts that extend this plan to support and enhance the testing process. The following artifacts make up part of this testing plan:

• Test cases

• Test scenarios

• Test procedures

• Testing matrix

• Defects matrix with corrective actions

• Error logs, bug reports, and/or screen captures (where feasible)

• System error logs, bug reports, and/or screen captures (where feasible)

• Webinar on the Linden Deployment


Test Cases

Based on the RFP, contract, and agreements between EasyMile and Smart Columbus, Table 6-1 outlines the functions of the system that are to be tested and/or validated. The columns under testing phase mark the testing phase(s) in which each test shall be conducted. Additional detail on how these functions will be tested is included in Chapter 7 and Chapter 8.

Table 6-1: CEAV System Test Cases

ID VERIFICATION METHOD TEST OBJECTIVE

TESTING PHASE FACTORY PRELIM. FINAL

CEAV-001-v01 Inspection Provide and maintain two EasyMile EZ10 electric shuttle vehicles manufactured after 2018 (these vehicles shall be newly manufactured).

x x

CEAV-002-v01 Demonstration Program the vehicles with speed for each road section and the presence of traffic signals, traffic signs (stop, yield, etc.), crosswalks, traffic circles, and stations.

Test Track (TT)4 x x

CEAV-003-v01 Demonstration Program the vehicles with an alert to indicate the vehicle will begin moving following a full stop. TT x x

CEAV-004-v01 Demonstration Demonstrate the procedure to immediately notify the City of any crashes or incidents related to transporting customers and/or involving the vehicle.

x x

CEAV-005-v01 Demonstration Provide a human operator with the ability to take manual control of the vehicle if deemed necessary. TT x x

CEAV-006-v01 Inspection Train and certify human operators to know how to operate the shuttles and communicate with and assist passengers. x

CEAV-007-v01 Test Ensure vehicles have sensors that can detect objects. x

4 Refers to EasyMile’s test track in Toulouse, France.

Chapter 6. Test Cases




CEAV-008-v01 Analyze Certify that the factory at which the vehicles are manufactured have strong quality assurance/quality control processes in place.

x

CEAV-009-v01 Inspection Have the ability to monitor information on the vehicles battery level. TT x x

CEAV-010-v01 Demonstration Perform a low-speed merge. TT x x

CEAV-011-v01 Demonstration Pull over to the side of the road. x x

CEAV-012-v01 Demonstration Move out of the travel lane (if applicable) and stop in order to service stop locations. TT x x

CEAV-013-v01 Demonstration Perform car following when approaching intersections. x

CEAV-014-v01 Demonstration Perform car following in stop and go traffic conditions. x x

CEAV-015-v01 Demonstration Navigate signalized intersections. TT

CEAV-016-v01 Demonstration Navigate unsignalized intersections. TT x

CEAV-017-v01 Demonstration Perform left and right turns. TT x x

CEAV-018-v01 Demonstration Enter and emerge from a stop-controlled traffic circle. x x

CEAV-019-v01 Demonstration Cross intersections with traffic speed limits up to 35 mph. x x

CEAV-020-v01 Demonstration Make appropriate right-of-way decisions when merging from a transit stop (pulling out when clear). x x

CEAV-021-v01 Demonstration Make appropriate right-of-way decisions at intersections (proceeding when it’s the vehicle’s turn). TT x x

CEAV-022-v01 Demonstration

Make appropriate right-of-way decisions when interacting with vulnerable road users (VRUs) (yielding for a VRU crossing the vehicle’s path, following a slower moving VRU, and overtaking a stopped VRU).

TT x x

CEAV-023-v01 Demonstration Detect and respond to encroaching oncoming vehicles (stop and move over, possibly prompt the onboard operator for guidance).

TT x x

CEAV-024-v01 Demonstration Detect stopped vehicles in the vehicle’s path. TT x x

CEAV-025-v01 Demonstration Pass stopped vehicles when safe (i.e., when opposing direction is clear). Note that passing when safe in this and TT x x





subsequent test cases may require a manual override prompted by the automated system.

CEAV-026-v01 Demonstration

Detect and respond to static obstacles by passing when safe. Static obstacles are generally movable but currently static obstacles such as parked vehicles, trash cans, etc. that have not been mapped.

TT x x

CEAV-027-v01 Demonstration Detect and respond to moving obstacles by following if they are slower moving and/or passing when safe. TT x x

CEAV-028-v01 Demonstration Interact safely and appropriately with emergency vehicles (likely by coming to a stop). x x

CEAV-029-v01 Demonstration Detect and respond to vulnerable road users in the vehicle’s projected travel path, including at intersections and crosswalks.

TT x x

CEAV-030-v01 Demonstration Detect and respond to vulnerable road users approaching the vehicle’s projected travel path, including at intersections and crosswalks.

TT x x

CEAV-031-v01 Demonstration Provide a safe distance from vehicles, pedestrians, bicyclists, and scooters on the side of the road. TT x x

CEAV-032-v01 Demonstration Provide a safe distance from vehicles, pedestrians, bicyclists, and scooters sharing the lane. TT x x

CEAV-033-v01 Demonstration Decrease speed when there is uncertainty regarding which action to take. TT x x

CEAV-034-v01 Demonstration Detect and respond to detours and other temporary changes in traffic patterns. x x

CEAV-035-v01 Analyze

Operate in rain, fog, and light snow conditions not deemed a weather emergency when such conditions are available at normal speed, stopping distance, object detection rate/accuracy, with no unexpected reactions (stopping,

x x





swerving, etc.) by the software that is caused by the weather condition.5

CEAV-036-v01 Demonstration Deploy a wheelchair accessible ramp. x x x

CEAV-037-v01 Test Provide real-time vehicle location information. x x

CEAV-038-v01 Test

Employ technology to count passenger boardings and alightings, with both location and time attributes, and report these on a daily basis through an API to the Operating System.

x

CEAV-039-v01 Demonstration Stop and open doors at designated locations to allow passengers to board and alight. x x x

CEAV-040-v01 Demonstration Stop and open doors if the vehicle has detected that there is an issue on board, through passenger input and/or secure override.

TT x x

CEAV-041-v01 Inspection Confirm the vehicles have multiple secure means of egress, in the event the primary exit is blocked and/or power failure occurs.

x x x

CEAV-042-v01 Demonstration Provide directional (i.e., northbound to St. Stephen’s or southbound to Linden Transit Center) information in audible and visual form to passengers.

x x

CEAV-043-v01 Test Provide free Wi-Fi to onboard passengers. x

5 Note that such conditions may not be available during the proposed testing dates. The vehicles shall be permitted to proceed to operations even if they have not yet been tested in these conditions, with the caveat that testing will recommence momentarily the first day such conditions become available. See Section 4.7.4 for additional detail.





CEAV-044-v01 Demonstration Equip vehicles with cameras capable of viewing the entirety of the passenger compartment and capable of capturing external video.6

x x x

CEAV-045-v01 Analyze Monitor local weather patterns and be aware of any approaching severe weather event or other conditions that may impact vehicle operations.

x

CEAV-046-v01 Analyze Meet a minimum headway of twelve minutes during hours in which service is provided. x

CEAV-047-v01 Analyze Collect required data and send this data to the Smart Columbus Operating System via an API in real-time or near real-time.7

x

CEAV-048-v01 Analyze Collect required data and send this data to the Smart Columbus Operating System via an API on a daily basis. x

CEAV-049-v01 Analyze Collect required data and send this data to the Smart Columbus Operating System via an API on a weekly basis. x

CEAV-050-v01 Inspection Provide a safe and secure location for overnight storage, charging, and routine maintenance x

CEAV-051-v01 Demonstration Show that each vehicle can take a charge from each charging port. x x

CEAV-052-v01 Demonstration Transport the vehicles between the service route and the facility. x x


6 Note that at this time, the cameras will simply be capable of providing footage in real-time to a remote location, and that this footage will not be recorded or retained for future use.

7 Note that the data sharing requirements, included in Appendix D, specify which data needs to be sent in real-time/near real-time, on a daily basis, or on a weekly basis.


Test Scenarios

Test scenarios are critically important to the acceptance of the system, as they test the system’s ability to meet the requirements holistically, end-to-end, from all active participant viewpoints (passenger, human operator, City of Columbus, and system administrator). Scenarios are made up of a series of test cases used to simulate the system in a real-world operational environment. This approach validates the system’s ability to meet the concepts established through the project’s Operational Concept and provides a decision gate for the production release of the service.

The scenarios outlined below are reflective of the capabilities the system must be able to perform in order to receive acceptance of the solution. The system capability and performance will be deemed acceptable provided all testing elements within a scenario successfully pass the test from each participant’s viewpoint. Each scenario must meet the stated outcome for each test case as detailed in the test procedures in Chapter 8 in order to be marked as pass. A range of closed course and real-world testing scenarios are described in this section to present how the proper functioning of the entire system with be tested.

Testing preconditions include the need for two EasyMile EZ10 shuttles that will be used for testing and eventual passenger service. Tests may also require other vehicles, test personnel, and other devices as appropriate.

The EasyMile EZ10 Acceptance Test Manual can be found in Appendix E and outlines the tests conducted at the factory.


Table 7-1: Acceptance Test Scenarios – Factory Acceptance Testing

FUNCTION SCENARIO DESCRIPTION SCENARIO DETAILS REQUIREMENTS TRACEABILITY

Factory Quality Management Audit

Ensure the factory in which the vehicles that will be delivered were manufactured is in good working condition and has strong quality assurance programs in place.

• Verify factory quality management program • Demonstrate emergency response procedures • Verify training procedures and requirements • Inspect the quality of the exact EZ10 shuttles

that will be delivered • See factory condition, signage, strategies, etc.,

first-hand • Determine best practices from prior

deployments and vehicle construction, such as emergency management procedures

• Verify training is conducted and is sufficient • Ensure vehicles that will be delivered have no

defects

CEAV-OPS-006-v01 CEAV-OPS-011-v01 CEAV-OPS-021-v01 CEAV-VEH-001-v01

Initial Vehicle Functional Verification

Demonstrate basic functions of a vehicle such as transitioning from automated to manual mode and object detection. Demonstrate overall system functionality on a test track if possible.

• Verify the proper condition of vehicle components to operate as designed

• Verify how vehicle functions are programmed, and how this can be modified for each deployment

• Demonstrate a manual override • Perform component testing for essential vehicle

components such as sensors • Demonstrate the ability of vehicles to operate as

intended • Record how vehicles are programmed • Verify a manual system override and the

security thereof • Verify that individual components are functional • Perform a test validation run using an actual

EZ10 shuttle on a test track

CEAV-VEH-011-v01 CEAV-VEH-012-v01 CEAV-VEH-013-v01 CEAV-VEH-014-v01 CEAV-VEH-015-v01 CEAV-VEH-016-v01 CEAV-VEH-017-v01 CEAV-VEH-018-v01 CEAV-VEH-021-v01 CEAV-VEH-022-v01 CEAV-VEH-023-v01 CEAV-VEH-024-v01 CEAV-VEH-025-v01 CEAV-VEH-026-v01 CEAV-VEH-027-v01 CEAV-VEH-028-v01 CEAV-VEH-029-v01 CEAV-VEH-030-v01 CEAV-VEH-032-v01

Chapter 7. Test Scenarios



CEAV-VEH-033-v01 CEAV-VEH-034-v01 CEAV-VEH-035-v01 CEAV-VEH-036-v01 CEAV-VEH-037-v01 CEAV-VEH-039-v01 CEAV-VEH-041-v01 CEAV-VEH-043-v01


Table 7-2: Acceptance Test Scenarios – Preliminary Acceptance Testing and Final Acceptance Testing


Route run

Perform a full loop of the service route that includes stopping at every stop to open and close doors, simulate boardings and alightings, and interact with mixed traffic.

• A series of route runs will comprise Preliminary and Final Acceptance Testing

• The vehicles must be able to perform the entire route, with minimal manual overrides as specified in Table 6-1

• Individual test cases will be conducted within these route runs, and may require additional equipment such as another vehicle or object for the vehicle under testing to detect

CEAV-VEH-011-v01 CEAV-VEH-012-v01 CEAV-VEH-013-v01 CEAV-VEH-014-v01 CEAV-VEH-015-v01 CEAV-VEH-016-v01 CEAV-VEH-017-v01 CEAV-VEH-018-v01 CEAV-VEH-019-v01 CEAV-VEH-020-v01 CEAV-VEH-021-v01 CEAV-VEH-022-v01 CEAV-VEH-023-v01 CEAV-VEH-024-v01 CEAV-VEH-025-v01 CEAV-VEH-026-v01 CEAV-VEH-027-v01 CEAV-VEH-028-v01 CEAV-VEH-029-v01




CEAV-VEH-030-v01 CEAV-VEH-031-v01 CEAV-VEH-032-v01 CEAV-VEH-033-v01 CEAV-VEH-034-v01 CEAV-VEH-035-v01 CEAV-VEH-036-v01 CEAV-VEH-037-v01 CEAV-VEH-038-v01 CEAV-VEH-039-v01 CEAV-VEH-040-v01 CEAV-VEH-041-v01 CEAV-VEH-042-v01 CEAV-VEH-043-v01

Passenger ride

Simulation of an entire passenger trip, from identifying a stop location, receiving real-time location information, boarding a vehicle, securely riding the vehicle, and alighting the vehicle at the intended stop.

• A member of the testing team can act as a passenger

• Will also test accessibility – volunteers can be solicited to test the vehicle and ensure it is compatible with their needs (mobility, vision, and hearing restrictions).

• The vehicle pulls over, opens doors, allows passengers to board and/or alight, closes doors, and indicates when it is ready to proceed. The vehicle will also demonstrate the ability to move in and out of mixed traffic.

CEAV-VEH-004-v01 CEAV-VEH-009-v01 CEAV-VEH-039-v01 CEAV-VEH-043-v01

Manual operator override

Operator can quickly and security intervene in vehicle operations when onboard the vehicle.

• Demonstrate the ability to quickly and safely perform a manual override

• Show that the method of vehicle intervention is secure (i.e., if it is wireless that it is over a secure network and that there are distance limitations)

CEAV-VEH-041-v01




Temporary change in traffic patterns

The vehicle detects that there are changes to normal route conditions and deviates from its normal path (with or without manual operator assistance).

• For example, navigating around a stopped vehicle

• SOPs should also consider the ability to deviate from the normal route in the event a road is closed

CEAV-VEH-037-v01 CEAV-VEH-042-v01

Detection of incoming inclement weather and decision whether to suspend service

The monitoring system has a method to determine when weather conditions may degrade to a point at which vehicle systems are either less effective or in danger of being damaged.

• Confirm that the vehicle can operate in some inclement weather, though not all

• Also assess after the fact whether the vehicle could have operated under the weather as it actually occurred

CEAV-OPS-014-v01 CEAV-VEH-038-v01 CEAV-DAT-019-v01

Incident response

An emergency situation, that may involve the vehicle or may not, occurs, and the vehicle responds by moving to a safe location and alerting authorities.

• See SOPs, validate in the tabletop exercise

CEAV-OPS-018-v01 CEAV-OPS-020-v01 CEAV-OPS-022-v01

Service start/end

The vehicle transitions to or from the storage facility to active service.

• Each vehicle will perform a full run of the route before offering rides to passengers

• A similar procedure will be in the SOPs for if the vehicle has to be taken out of service due to an emergency or approaching inclement weather

CEAV-OPS-004-v01

Vehicle storage

The vehicle transitions to or from active service to the storage facility.

• At the end of the service period, each vehicle is able to enter the storage facility and be stored in a secure location

• The storage facility itself is also inspected, including charging (the vehicle is attached to a charging station and sufficiently charging to support service)

CEAV-OPS-004-v01 CEAV-OPS-013-v01

Preventive maintenance

Preventive maintenance is performed on a predetermined

• See SOPs • Project parties agree on the appropriate

schedule

CEAV-OPS-004-v01 CEAV-OPS-005-v01 CEAV-OPS-007-v01




schedule as outlined in the SOPs.

CEAV-OPS-012-v01

Reactive maintenance

After a defect is detected or an incident occurs, a vehicle is pulled from service to perform maintenance.

• See SOPs • In this scenario, project partners would meet to

determine how long the vehicle may be out of service and what the next steps would be

CEAV-OPS-004-v01



Test Procedures

This Chapter provides additional detail on how the functions/test cases presented previously will be tested and evaluated by the Smart Columbus Project team. The Chapter is broken down into three subsections, one for each of the testing phases, because many test cases will be demonstrated in a different way depending on the phase.

8.1. FACTORY ACCEPTANCE TESTING During Factory Acceptance Testing, each vehicle will conduct at least 50 km (31 mi) of testing at the factory, with three stops for each loop of the test track route. Vehicle capabilities will be demonstrated throughout this distance of testing, with stop-based capabilities demonstrated at every stop. The minimum number of tests for capabilities that do not occur regularly throughout operations, such as deploying the ramp, is specified in the evaluation criteria as necessary.

Table 8-1: Factory Acceptance Test 1

Test Number and Title FAT1: Vehicle Manufacturing Function(s) Tested CEAV-001-v01

Test Objective Ensure that the vehicles that will be delivered to Columbus are newly manufactured and electric-powered.

Equipment & Setting/Environment Two EZ10 shuttles in EasyMile’s factory and test track Personnel Required System Owner, Test Manager, Test Conductor

Procedure

1. System Owner obtains two EZ10 shuttles. 2. System Owner obtains paperwork for two EZ10 shuttles. 3. System Owner verifies that the paperwork certifies that the

shuttles are 2018 or newer. 4. System Owner verifies that the shuttle VINs and general

condition match with the paperwork. 5. System Owner verifies that the NHTSA exemption

approval matches the shuttle VINs and is complete. 6. System Owner verifies that the shuttles are operating on

the same software release for all tests.

Evaluation Criteria Considered met if paperwork for both vehicles shows the correct and expected information, and a visual inspection confirms there are no defects or signs of use on the shuttles. Preventive maintenance will be scheduled per the SOPs.



Test Number and Title FAT2: Route Programming Function(s) Tested CEAV-002-v01

Test Objective Ensure the vehicles can be programmed to operate in an environment similar to the eventual route conditions.

Chapter 8. Test Procedures


Test Number and Title FAT2: Route Programming Equipment & Setting/Environment Two EZ10 shuttles in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle

2. Test Conductor assigns the mission to the vehicle to follow the route

3. EZ10 shuttle begins the mission 4. Test Manager verifies the shuttle is following the assigned

route

Evaluation Criteria

Met if not a single critical failure (as defined in Section 4.2) is experienced. Requiring a human operator to regularly perform a manual check before telling the vehicle to proceed is acceptable, assuming the vehicle proceeds on its own once it is told the manual check is complete.



Test Number and Title FAT3: Movement Alert Function(s) Tested CEAV-003-v01

Test Objective Ensure the vehicles have an indicator to alert passengers when the vehicle will begin moving following a stop.

Equipment & Setting/Environment Two EZ10 shuttles in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. Test Manager verifies an audible alert is presented inside

and outside the shuttle cabin to alert others that the vehicle is about to being moving

5. Test Manager confirms this audible alert is activated when the vehicle begins moving after a stop

Evaluation Criteria Met if the alert is activated at 100% of all stop departures. Source: City of Columbus


Test Number and Title FAT4: Manual Override Function(s) Tested CEAV-005-v01 Test Objective Ensure a manual override is possible. Equipment & Setting/Environment Two EZ10 shuttles in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure 1. Test Conductor loads a pre-programmed route onto an

EZ10 shuttle 2. Test Conductor assigns the mission to the vehicle to follow

the route



Test Number and Title FAT4: Manual Override 3. EZ10 shuttle begins the mission 4. At a desired point during the mission, Test Conductor

takes over manual control of the vehicle 5. Test Manager verifies that the shuttle is under manual

control and that the internal screen and external LED light bar both indicate that manual control is engaged

Evaluation Criteria

Test Manger may request that the Test Conductor demonstrate a manual override at any location (which will be induced by placing a temporary obstacle in the shuttle’s path). In addition, if a manual override becomes necessary for any reason during testing, this procedure will be assessed as it occurs. Met if the transition between automated mode and the operator is considered smooth by the testing team; that is, it begins with a prompt by the vehicle system and the onboard operator is easily able to access the controls and knows what to do.



Test Number and Title FAT5: Sensors Function(s) Tested CEAV-007-v01

Test Objective Ensure the object detection sensors have been installed on the vehicles and are functional.


Procedure

1. Test Conductor ensures the vehicle is on 2. Test Manager monitors the sensor visualization capabilities

(that are not presented to the end user) 3. Test Conductor has another tester walk around the vehicle 4. Test Manager verifies the person is visualized based on

the sensor readings Evaluation Criteria Met if 100% of objects shown to sensors are detected.



Test Number and Title FAT6: Factory Audit Function(s) Tested CEAV-008-v01

Test Objective Ensure the factory where the vehicles were manufactured meets the City’s expectations.

Equipment & Setting/Environment EasyMile’s factory Personnel Required System Owner, Test Manager, Test Conductor

Procedure 1. During the factory walk through, System Owner asks

questions, makes observations, and takes note of these processes.

Evaluation Criteria Met if conditions are considered satisfactory to create a reliable technology.





Test Number and Title FAT7: Battery Level Monitoring Function(s) Tested CEAV-009-v01

Test Objective Ensure that EasyMile is able to monitor the current battery level at all times.

Equipment & Setting/Environment Two EZ10 shuttles, EZ Fleet in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure

1. Test Conductor points out where the battery monitor is located on a stationary vehicle.

2. As testing is conducted, Test Manager checks battery level periodically and confirms it is decreasing.

3. After the vehicle returns to the facility, the Test Conductor plugs the vehicle in and demonstrates charging procedure.

4. Test Manager confirms that a charging vehicle shows its battery level increasing.

5. Test Manager verifies that the battery information is provided to EZ Fleet software and that this information matches with the information provided by the indicator in the vehicle.

Evaluation Criteria Met if battery performance is sufficient to support operations and if SOPs include specifications on the level at which a vehicle will be taken out of service.



Test Number and Title FAT8: Low-Speed Merge Function(s) Tested CEAV-010-v01 Test Objective Demonstrate a low-speed merge.

Equipment & Setting/Environment Two EZ10 shuttles, one additional vehicle in EasyMile’s factory and test track

Personnel Required Test Manager, Test Conductor

Procedure 1. Test Conductor provides the vehicle with confirmation that

it is safe to proceed at each stop departure. 2. Test Manager verifies that the vehicle proceeds when the

roadway is clear, and that it waits if it is not clear.

Evaluation Criteria

Met if vehicle performs the maneuver without manual override for 100% of stop departures (operator providing vehicle with confirmation that it is safe to proceed is acceptable, but the operator should not use the remote to steer the vehicle unless an obstacle is blocking the path).





Test Number and Title FAT9: Serving a Stop Location Function(s) Tested CEAV-012-v01

Test Objective Demonstrate moving out of the travel lane or stopping on the side of the road to serve a stop location.


Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle that includes at least four stops


3. EZ10 shuttle begins the mission 4. Test Manager verifies that the shuttle pulls over to serve

each of the stops as programmed. This includes moving out of the travel lane if there is space available or stopping on the side of the road, depending on the stop layout.

5. EZ10 shuttle dwells for the programmed dwell time, or longer if there continue to be passengers boarding or alighting after this time

6. Test Conductor performs FAT8

Evaluation Criteria Met if vehicle performs the maneuver without manual override for 100% of stop arrivals.



Test Number and Title FAT10: Signalized Intersection Function(s) Tested CEAV-015-v01 Test Objective Demonstrate how a signalized intersection would be navigated. Equipment & Setting/Environment Two EZ10 shuttles in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. When the vehicle approaches the signalized intersection,

Test Manager verifies that it comes to a stop on red signal or proceeds on a green signal

5. If the vehicle arrived on red, Test Manager verifies that it proceeds on green when the indication changes.

Evaluation Criteria Met if record of data corresponds with actual signal state for all times it is available. For Factory Acceptance Testing, met if vehicle can navigate through a connected signalized intersection.





Test Number and Title FAT11: Unsignalized Intersection Function(s) Tested CEAV-016-v01 Test Objective Demonstrate how an unsignalized intersection is navigated. Equipment & Setting/Environment Two EZ10 shuttles in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. When the vehicle approaches an unsignalized

intersection, the vehicle comes to a stop if it is on the stop-controlled approach

5. Test Conductor (onboard operator) provides the vehicle with confirmation to proceed when it is safe to do so (i.e., that there are no other vehicles at or approaching the intersection or the EZ10 arrived at the intersection before any other vehicles and the operator perceives that all other vehicles will stop, will remain stopped, or will not make a conflicting movement)

6. EZ10 shuttle proceeds through the intersection

Evaluation Criteria This will be demonstrated at each intersection approach throughout testing and will include both two-way and four-way stop controlled intersections.



Test Number and Title FAT12: Left and Right Turns Function(s) Tested CEAV-017-v01 Test Objective Demonstrate a left and a right turn. Equipment & Setting/Environment Two EZ10 shuttles in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. Test Manager verifies that the vehicle can make both a left

turn and a right turn along the route Evaluation Criteria This will be demonstrated as appropriate throughout testing.



Test Number and Title FAT13: Intersection Right-of-Way Function(s) Tested CEAV-021-v01



Test Number and Title FAT13: Intersection Right-of-Way

Test Objective Demonstrate making right-of-way decisions at an unsignalized intersection.



Procedure



3. EZ10 shuttle begins the mission 4. When the vehicle approaches an unsignalized

intersection, the vehicle comes to a stop if it is on the stop-controlled approach

5. One additional vehicle also comes to a stop at the same intersection, either right before or right after the EZ10

6. Test Conductor (onboard operator) makes appropriate right-of-way decisions and provides the EZ10 with confirmation to proceed when it is the vehicle’s turn


Evaluation Criteria

This will be demonstrated at least 5 times during Factory Acceptance Testing. The appropriate maneuver is for the vehicle to proceed when it’s the vehicle’s turn. If all approaches are stop-controlled, it will proceed based on order of arrival. If there is no stop-control on its approach, it will continue moving through the intersection.



Test Number and Title FAT14: VRU Right-of-Way Function(s) Tested CEAV-022-v01

Test Objective Demonstrate making right-of-way decisions when interacting with VRUs.

Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a VRU (pedestrian, bicyclist, scooter) in EasyMile’s factory and test track


Procedure



3. EZ10 shuttle begins the mission 4. VRU stands in the road at a point along the route or

approaches the vehicle’s path (and is within the programmed detection area when the vehicle arrives)

5. Test Manager verifies the shuttle stops as it approaches the VRU

6. VRU walks or rides along the programmed route 7. EZ10 shuttle follows at a safe speed 8. EZ10 passes the VRU when it is safe to do so (by

engaging manual mode)



Test Number and Title FAT14: VRU Right-of-Way 9. Once the VRU is out of the path, verify normal operation

resumes

Evaluation Criteria

This will be demonstrated at least 10 times for a pedestrian VRU and at least 10 times for a bicyclist or scooter VRU during Factory Acceptance Testing. The appropriate maneuvers include yielding for a VRU crossing the vehicle’s path, following a slower moving VRU, and overtaking a stopped VRU (using manual mode). See FAT20 and FAT21 for variants and additional tests.



Test Number and Title FAT15: Encroaching Oncoming Vehicles Function(s) Tested CEAV-023-v01

Test Objective Demonstrate detecting and responding to encroaching oncoming vehicles.



Procedure



3. EZ10 shuttle begins the mission 4. The additional vehicle approaches the shuttle near the

assigned path 5. Test Manager verifies the shuttle reacts appropriately,

either slowing or stopping based on the closeness of the approaching vehicle

Evaluation Criteria This will be demonstrated at least 5 times during Factory Acceptance Testing. The appropriate maneuvers include slowing down, stopping, and possibly prompting the onboard operator for guidance or to take control.



Test Number and Title FAT16: Detect Stopped Vehicle Function(s) Tested CEAV-024-v01 Test Objective Demonstrate detecting stopped vehicles in the path.



Procedure



3. EZ10 shuttle begins the mission



Test Number and Title FAT16: Detect Stopped Vehicle 4. The additional vehicle is parked in the road in EZ10’s

programmed path 5. Test Manager verifies the shuttle slows to a stop as it

approaches the vehicle

Evaluation Criteria This will be demonstrated at least 5 times during Factory Acceptance Testing. The appropriate maneuver is for the vehicle to stop and prompt for additional guidance.



Test Number and Title FAT17: Pass Stopped Vehicle Function(s) Tested CEAV-025-v01 Test Objective Demonstrate passing a stopped vehicle when safe



Procedure 1. Test Conductor performs FAT16 2. Test Conductor engages manual mode and passes the

stopped vehicle when it is safe to do so

Evaluation Criteria This will be demonstrated at least 5 times during Factory Acceptance Testing. The appropriate maneuver is for the vehicle to wait until the opposing direction is clear and then pass.



Test Number and Title FAT18: Static Obstacles Function(s) Tested CEAV-026-v01 Test Objective Demonstrate detecting and responding to static obstacles.

Equipment & Setting/Environment Two EZ10 shuttles, static obstacles in EasyMile’s factory and test track


Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor loads the sensor visualization 5. Test Conductor presents objects 6. Test Manager verifies that static objects are detected by

the sensors 7. Test Manager verifies that the shuttle proceeds only when

clear

Evaluation Criteria

This will be demonstrated at least 5 times during Factory Acceptance Testing for each of the following objects:

• Parked cars • Trees



Test Number and Title FAT18: Static Obstacles • Curbs • Buildings



Test Number and Title FAT19: Moving Obstacles Function(s) Tested CEAV-027-v01 Test Objective Demonstrate detecting and responding to moving obstacles.

Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a VRU in EasyMile’s factory and test track


Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor loads the sensor visualization 5. Test Manager verifies that moving objects are detected by


clear

Evaluation Criteria

This will be demonstrated at least 5 times during Factory Acceptance Testing for each of the following objects:

• Pedestrian • Scooter • Bicyclist • Ball or cone (object below knee height) • Another vehicle



Test Number and Title FAT20: VRU in Travel Path Function(s) Tested CEAV-029-v01

Test Objective Demonstrate detecting and responding to a VRU in the vehicle’s travel path.



Procedure

1. Test Conductor performs FAT14 for a VRU in the vehicle’s path

2. Test Manager verifies that the EZ10 stopped for the VRU either using a soft stop or an e-stop

3. Test Manager verifies that the vehicle’s sensors detected the VRU



Test Number and Title FAT20: VRU in Travel Path

Evaluation Criteria This will be demonstrated at least 5 times for a pedestrian VRU and at least 5 times for a bicyclist or scooter VRU during Factory Acceptance Testing.



Test Number and Title FAT21: VRU approaching Travel Path Function(s) Tested CEAV-030-v01

Test Objective Demonstrate detecting and responding to a VRU approaching the vehicle’s travel path.



Procedure

1. Test Conductor performs FAT14 for a VRU approaching the vehicle’s path (rather than already in it)



Evaluation Criteria This will be demonstrated at least 5 times for a pedestrian VRU and at least 5 times for a bicyclist or scooter VRU during Factory Acceptance Testing.



Test Number and Title FAT22: Safe Distance to Side of Road Function(s) Tested CEAV-031-v01

Test Objective Demonstrate providing a safe distance from objects on the side of the road.

Equipment & Setting/Environment Two EZ10 shuttles, static obstacle in EasyMile’s factory and test track


Procedure



3. EZ10 shuttle begins the mission 4. Test Manager verifies that the shuttle is programmed to

maintain a distance from object along the road 5. Test Manager verifies if an object is closer than expected,

the shuttle reacts by slowing or stopping

Evaluation Criteria

This will be demonstrated at least 5 times throughout Factory Acceptance Testing. A safe distance is one that does not risk a collision, and in which the other object does not need to take evasive action. This safe distance will be programmed into the vehicle system prior to testing.





Test Number and Title FAT23: Safe Distance Sharing a Lane Function(s) Tested CEAV-032-v01

Test Objective Demonstrate providing a safe distance from objects sharing the lane.

Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a bicyclist or scooter in EasyMile’s factory and test track


Procedure



3. EZ10 vehicle begins the mission 4. Bicyclist or scooter rides in front of the shuttle 5. Test Manager verifies that the shuttle follows at the same

rate of speed or less from a safe distance, or that if it passes the bicyclist or scooter (in manual mode) it does so at a safe distance.

Evaluation Criteria

This will be demonstrated at least 5 times during Factory Acceptance Testing. A safe distance is one that does not risk a collision, and in which the other object does not need to take evasive action. This safe distance will be programmed into the vehicle system prior to testing. Note that it is also acceptable for the vehicle to continue following at a slow speed.



Test Number and Title FAT24: Decreasing Speed due to Uncertainty Function(s) Tested CEAV-033-v01

Test Objective Demonstrate decreasing speed when there is uncertainty regarding which action to take due to an unexpected object.

Equipment & Setting/Environment Two EZ10 shuttles, traffic cone in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor uses an object to encroach on the shuttle

path or uses a traffic cone to provide an unexpected situation

5. Test Manager verifies that the vehicle slows down and prompts the operator for guidance due to the presence of this unexpected object

Evaluation Criteria This will be demonstrated at least 5 times at different points throughout the route.





Test Number and Title FAT25: Deploy Ramp Function(s) Tested CEAV-036-v01

Test Objective Demonstrate that vehicles can deploy a wheelchair accessible ramp.


Procedure

1. Test Conductor pushes the blue button on the interior or exterior of the shuttle

2. Test Manager verifies the shuttle deploys the ramp automatically, prior to opening the door

3. Test Conductor pushes the blue button again 4. Test Manger verifies the shuttle first closes the door and

then retracts the ramp automatically

Evaluation Criteria This will be demonstrated at least 5 times during Factory Acceptance Testing.



Test Number and Title FAT26: Stop and Open Doors Function(s) Tested CEAV-039-v01

Test Objective Demonstrate stopping and opening doors to allow passengers to board and alight.


Procedure



3. EZ10 shuttle begins the mission 4. Test Manager verifies the shuttle doors open when

pushing the door button to board the vehicle 5. Test Manager verifies the shuttle doors open when

pushing the door button to alight the vehicle

Evaluation Criteria This will be demonstrated at least 5 times at each stop location throughout Factory Acceptance Testing.



Test Number and Title FAT27: Stop and Open Doors in Emergency Function(s) Tested CEAV-040-v01



Test Number and Title FAT27: Stop and Open Doors in Emergency

Test Objective Demonstrate stopping and opening doors if the vehicle has detected that there is an issue on board, through passenger input and/or secure override.


Procedure



3. EZ10 vehicle begins the mission 4. Test Conductor presses the e-stop button to bring the

vehicle to a stop 5. Test Conductor presses the door release button to release

the doors 6. Test Manager verifies the doors can be manually opened

Evaluation Criteria This will be demonstrated at least 5 times during Factory Acceptance Testing.



Test Number and Title FAT28: Multiple Means of Egress Function(s) Tested CEAV-041-v01 Test Objective Demonstrate that vehicles have multiple means of egress. Equipment & Setting/Environment Two EZ10 shuttles in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure

1. Test Manager verifies that there is an emergency door release button.

2. Test Conductor pushes the button 3. Test Manager verifies it releases the doors and the doors

can be opened manually 4. Test Manager verifies a safety hammer is located in the

vehicle

Evaluation Criteria The vehicle will be inspected for its door and a safety hammer and other potential means of egress.



Test Number and Title FAT29: Cameras Function(s) Tested CEAV-044-v01

Test Objective Demonstrate that vehicles are equipped with cameras capable of viewing internal and external to the vehicle.

Equipment & Setting/Environment Two EZ10 shuttles, EZ Fleet in EasyMile’s factory and test track Personnel Required Test Manager, Test Conductor

Procedure 1. Test Manager visually inspects cameras are installed in the vehicle



Test Number and Title FAT29: Cameras 2. Test Manager verifies on EZ Fleet that the cameras can be

monitored

Evaluation Criteria The testing team will view the web-based fleet management system, EZ Fleet, on two occasions and view all camera angles.


8.2. PRELIMINARY ACCEPTANCE TESTING During Preliminary Acceptance Testing, each vehicle will first conduct at least 20 runs of a closed course route. Once vehicle performance is considered satisfactory, each vehicle will then also conduct at least 5 runs of the service route on open roads. Vehicle capabilities will be demonstrated throughout these testing runs. The minimum number of tests for capabilities that do not occur regularly throughout operations, such as deploying the ramp, is specified in the evaluation criteria as necessary.

When the personnel required includes the Test Manager, this may at times be his or her designee.

Table 8-30: Preliminary Acceptance Test 1

Test Number and Title PAT1: Vehicle Manufacturing Function(s) Tested CEAV-001-v01

Test Objective Ensure that the vehicles that will be delivered to Columbus are newly manufactured and electric-powered.

Equipment & Setting/Environment Two EZ10 shuttles Personnel Required System Owner, Test Manager, Test Conductor

Procedure

1. System Owner obtains two EZ10 shuttles. 2. System Owner obtains paperwork for two EZ10 shuttles. 3. System Owner verifies that the paperwork certifies that the

shuttles are 2018 or newer. 4. System Owner verifies that the shuttle VINs and general

condition match with the paperwork. 5. System Owner verifies that the NHTSA exemption

approval matches the shuttle VINs and is complete. 6. System Owner verifies that the shuttles are operating on

the same software release for all tests.

Evaluation Criteria Considered met if paperwork for both vehicles shows the correct and expected information, and a visual inspection confirms there are no defects or signs of use on the shuttles. Preventive maintenance will be scheduled per the SOPs.



Test Number and Title PAT2: Route Programming Function(s) Tested CEAV-002-v01

Test Objective Ensure the vehicles can be programmed to operate on the route conditions in Columbus.

Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads Personnel Required Test Manager, Test Conductor



Test Number and Title PAT2: Route Programming

Procedure

1. Test Conductor loads a pre-programmed route that is designated for the PAT onto an EZ10 shuttle



route

Evaluation Criteria

Met if not a single critical failure (as defined in Section 4.2) is experienced throughout Preliminary Acceptance Testing. Requiring a human operator to regularly perform a manual check before telling the vehicle to proceed is acceptable, assuming the vehicle proceeds on its own once it is told the manual check is complete.



Test Number and Title PAT3: Movement Alert Function(s) Tested CEAV-003-v01

Test Objective Ensure the vehicles have an indicator to alert passengers when the vehicle will begin moving following a stop.


Procedure



3. EZ10 shuttle begins the mission 4. Test Manger verifies an audible alert is presented inside





Test Number and Title PAT4: Incident Notification Function(s) Tested CEAV-004-v01

Test Objective Ensure there are procedures to notify the City in the event of an incident involving the vehicles and/or passengers.

Equipment & Setting/Environment Staff and stakeholders at tabletop meeting Personnel Required Test Manager, Test Conductor, Stakeholders

Procedure 1. Test Conductor prepares SOPs that include this procedure 2. Test Manager and Stakeholders inspect the SOPs to verify

an incident notification procedure is included and acceptable



Test Number and Title PAT4: Incident Notification 3. Test Conductor participates in a tabletop meeting with Test

Manager and Stakeholders to demonstrate the procedure, address any issues, and solicit stakeholder feedback

4. Test Conductor updates the SOPs as appropriate

Evaluation Criteria Met if there is concurrence by Stakeholders that the approach is complete.



Test Number and Title PAT5: Manual Override Function(s) Tested CEAV-005-v01 Test Objective Ensure a manual override is possible. Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. At a desired point during the mission, Test Conductor takes

over manual control of the vehicle 5. Test Manager verifies that the shuttle is under manual


Evaluation Criteria




Test Number and Title PAT6: Battery Level Monitoring Function(s) Tested CEAV-009-v01

Test Objective Ensure that EasyMile is able to monitor the current battery level at all times.

Equipment & Setting/Environment Two EZ10 shuttles, EZ Fleet on closed course and open roads Personnel Required Test Manager, Test Conductor

Procedure 1. Test Conductor points out where the battery monitor is

located on a stationary vehicle. 2. As testing is conducted, Test Manager checks battery level

periodically and confirms it is decreasing.



Test Number and Title PAT6: Battery Level Monitoring 3. After the vehicle returns to the facility, the Test Conductor

plugs the vehicle in and demonstrates charging procedure. 4. Test Manager confirms that a charging vehicle shows its

battery level increasing. 5. Test Manager verifies that the battery information is

provided to EZ Fleet software and that this information matches with the information provided by the indicator in the vehicle.

Evaluation Criteria Met if battery performance aligns with specs, and if SOPs include specifications on the level at which a vehicle will be taken out of service.



Test Number and Title PAT7: Low-Speed Merge Function(s) Tested CEAV-010-v01 Test Objective Demonstrate a low-speed merge. Equipment & Setting/Environment Two EZ10 shuttles, one additional vehicle on open roads Personnel Required Test Manager, Test Conductor

Procedure 1. Test Conductor provides the vehicle with confirmation that it

is safe to proceed at each stop departure. 2. Test Manager verifies that the vehicle proceeds when the


Evaluation Criteria

Met if vehicle performs the maneuver without manual override for 100% of stop departures (operator providing vehicle with confirmation that it is safe to proceed is acceptable, but the operator should not use the remote to steer the vehicle unless an obstacle is blocking the path).



Test Number and Title PAT8: Pulling Over Function(s) Tested CEAV-011-v01 Test Objective Demonstrate pulling over to the side of the road. Equipment & Setting/Environment Two EZ10 shuttles on open roads Personnel Required Test Manager, Test Conductor

Procedure

1. Test Conductor decides the EZ10 shuttle should pull over to the side of the road in the interest of safety, because of an incident onboard, an unsafe vehicle on the roadway, an approaching emergency vehicle, or just to simulate the procedure during testing, etc.

2. Test Conductor engages manual mode 3. Test Manager verifies that the vehicle pulls from the lane of

travel to the side of the road and stops

Evaluation Criteria This will be demonstrated at least 5 times during Preliminary Acceptance Testing.





Test Number and Title PAT9: Serving a Stop Location Function(s) Tested CEAV-012-v01


Equipment & Setting/Environment Two EZ10 shuttles on open roads Personnel Required Test Manager, Test Conductor

Procedure






6. Test Conductor performs PAT7

Evaluation Criteria This will be demonstrated at each stop arrival. Met if vehicle performs the maneuver without manual override for 100% of stop arrivals. Test can be deferred if an object is blocking the stop location.



Test Number and Title PAT10: Car Following in Stop and Go Traffic Function(s) Tested CEAV-014-v01 Test Objective Demonstrate car following in stop and go traffic conditions.

Equipment & Setting/Environment Two EZ10 shuttles, one additional vehicle on closed course and open roads


Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor has the vehicle lead the shuttle on the route 5. Test Conductor has the vehicle brake and go to simulate

stop and go traffic conditions

Evaluation Criteria This will be demonstrated by using another test vehicle (closed course) or in mixed traffic (open roads) at least 5 times during Preliminary Acceptance Testing.





Test Number and Title PAT11: Left and Right Turns Function(s) Tested CEAV-017-v01 Test Objective Demonstrate a left and a right turn. Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads Personnel Required Test Manager, Test Conductor

Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle that has left and right turn movements



turn and a right turn along the route Evaluation Criteria This will be demonstrated as appropriate along the route.



Test Number and Title PAT12: Traffic Circle Function(s) Tested CEAV-018-v01

Test Objective Demonstrate entering and emerging from a stop-controlled traffic circle.

Equipment & Setting/Environment Two EZ10 shuttles, one additional vehicle on open roads Personnel Required Test Manager, Test Conductor

Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle that has a traffic circle as part of the route


3. EZ10 shuttle begins the mission 4. Test Manager verifies that the vehicle comes to a stop at

the entrance to the traffic circle 5. At the entrance to the traffic circle, the Test Conductor

(onboard operator) makes appropriate right-of-way decisions and provides the EZ10 with confirmation to proceed when it is the vehicle’s turn

6. EZ10 shuttle proceeds through the traffic circle 7. Test Manager verifies that the vehicle enters the traffic circle

when it is safe and clear of opposing traffic 8. Test Manager verifies that the vehicle exits the traffic circle

at the programmed location Evaluation Criteria This will be demonstrated as appropriate along the route.



Test Number and Title PAT13: Crossing Intersections Function(s) Tested CEAV-019-v01



Test Number and Title PAT13: Crossing Intersections Test Objective Demonstrate crossing intersections. Equipment & Setting/Environment Two EZ10 shuttles on open roads Personnel Required Test Manager, Test Conductor

Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle that has an intersection as part of the route


3. EZ10 shuttle begins the mission 4. Each time the vehicle reaches an intersection, the Test

Conductor (onboard operator) makes the decision on when it is safe to cross and provides the EZ10 with confirmation to proceed when it is safe

5. Test Manager verifies that the vehicle safety crosses the intersection when no opposing traffic is present

Evaluation Criteria This will be demonstrated as appropriate along the route. Source: City of Columbus


Test Number and Title PAT14: Merging from a Transit Stop Function(s) Tested CEAV-020-v01

Test Objective Demonstrate appropriate right-of-way decisions when merging from a transit stop.


Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle with at least four stops


3. EZ10 begins the mission 4. At each stop departure, the Test Conductor (onboard

operator) provides the shuttle with confirmation to proceed after stop operations (boarding and alighting) have been completed and once the roadway is clear

5. Test Manager verifies that the vehicle proceeds when the roadway is clear, and that it waits if it is not clear

Evaluation Criteria This will be demonstrated at each stop departure. The appropriate maneuver is waiting until it is clear to pull out.



Test Number and Title PAT15: Intersection Right-of-Way Function(s) Tested CEAV-021-v01


Equipment & Setting/Environment Two EZ10 shuttles on open roads



Test Number and Title PAT15: Intersection Right-of-Way Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. When the vehicle approaches an unsignalized intersection,

the vehicle comes to a stop if it is on the stop-controlled approach



7. EZ10 shuttle proceeds through the intersection if no vehicles are approaching the intersection

Evaluation Criteria

This will be demonstrated at least 5 times during Preliminary Acceptance Testing. The appropriate maneuver is for the vehicle to proceed when it’s the vehicle’s turn. If all approaches are stop-controlled, it will proceed based on order of arrival. If there is no stop-control on its approach, it will continue moving through the intersection.



Test Number and Title PAT16: VRU Right-of-Way Function(s) Tested CEAV-022-v01


Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a VRU (pedestrian, bicyclist, scooter) on closed course


Procedure



3. EZ10 shuttle begins the mission 4. VRU stands in the road at a point along the route or



6. VRU walks or rides along the programmed route 7. EZ10 shuttle follows at a safe speed 8. EZ10 passes the VRU when it is safe to do so (by engaging

manual mode) 9. Once the VRU is out of the path, verify normal operation

resumes



Test Number and Title PAT16: VRU Right-of-Way

Evaluation Criteria

This will be demonstrated at least 10 times for a pedestrian VRU and at least 10 times for a bicyclist or scooter VRU during Preliminary Acceptance Testing. The appropriate maneuvers include yielding for a VRU crossing the vehicle’s path, following a slower moving VRU, and overtaking a stopped VRU (using manual mode). See PAT24 and PAT25 for variants and additional tests.



Test Number and Title PAT17: Encroaching Oncoming Vehicles Function(s) Tested CEAV-023-v01




Procedure




assigned path 5. Test Manager verifies the shuttle reacts appropriately, either

slowing or stopping based on the closeness of the approaching vehicle

Evaluation Criteria This will be demonstrated at least 5 times during Preliminary Acceptance Testing. The appropriate maneuvers include slowing down, stopping, and possibly prompting the onboard operator for guidance or to take control.



Test Number and Title PAT18: Detect Stopped Vehicle Function(s) Tested CEAV-024-v01 Test Objective Demonstrate detecting stopped vehicles in the path.



Procedure



3. EZ10 shuttle begins the mission 4. The additional vehicle is parked in the road in EZ10’s

programmed path



Test Number and Title PAT18: Detect Stopped Vehicle 5. Test Manager verifies the shuttle slows to a stop as it


Evaluation Criteria This will be demonstrated at least 5 times during Preliminary Acceptance Testing. The appropriate maneuver is for the vehicle to stop and prompt for additional guidance.



Test Number and Title PAT19: Pass Stopped Vehicle Function(s) Tested CEAV-025-v01 Test Objective Demonstrate passing a stopped vehicle when safe



Procedure 1. Test Conductor performs PAT19 2. Test Conductor engages manual mode and passes the


Evaluation Criteria This will be demonstrated at least 5 times during Preliminary Acceptance Testing. The appropriate maneuver is for the vehicle to wait until the opposing direction is clear and then pass.



Test Number and Title PAT20: Static Obstacles Function(s) Tested CEAV-026-v01 Test Objective Demonstrate detecting and responding to static obstacles. Equipment & Setting/Environment Two EZ10 shuttles on closed course Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor loads the sensor visualization 5. Test Conductor presents objects 6. Test Manager verifies that static objects are detected by the

sensors 7. Test Manager verifies that the shuttle proceeds only when

clear

Evaluation Criteria

This will be demonstrated at least 5 times during Preliminary Acceptance Testing for each of the following objects:

• Parked cars • Trees • Curbs • Buildings • Garbage cans



Test Number and Title PAT20: Static Obstacles • Piles of leaves (when available, similar to weather testing) • Piles of snow (when available, similar to weather testing)



Test Number and Title PAT21: Moving Obstacles Function(s) Tested CEAV-027-v01 Test Objective Demonstrate detecting and responding to moving obstacles. Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a VRU on closed course Personnel Required Test Manager, Test Conductor

Procedure





clear

Evaluation Criteria

This will be demonstrated at least 5 times during Preliminary Acceptance Testing for each of the following objects:




Test Number and Title PAT22: Emergency Vehicle Interaction Function(s) Tested CEAV-028-v01

Test Objective Ensure there are procedures to ensure safe and appropriate interactions with emergency vehicles.

Equipment & Setting/Environment Staff and stakeholders at tabletop meeting Personnel Required Test Manager, Test Conductor, Stakeholders

Procedure

1. Test Conductor prepares SOPs that include this procedure 2. Test Manager and Stakeholders inspect the SOPs to verify

that interactions with emergency vehicles are included and the procedure is acceptable (it will likely include a manual override)

3. Test Conductor participates in a tabletop meeting with Test Manager and Stakeholders to demonstrate the procedure, address any issues, and solicit stakeholder feedback

4. Test Conductor updates the SOPs as appropriate



Test Number and Title PAT22: Emergency Vehicle Interaction

Evaluation Criteria Met if there is concurrence by Stakeholders that the approach is complete. The appropriate maneuver is for the vehicle to stop and prompt the onboard operator for additional guidance.



Test Number and Title PAT23: VRU in Travel Path Function(s) Tested CEAV-029-v01




Procedure

1. Test Conductor performs PAT17 2. Test Manager verifies that the EZ10 stopped for the VRU

either using a soft stop or an e-stop 3. Test Manager verifies that the vehicle’s sensors detected

the VRU

Evaluation Criteria This will be demonstrated at least 5 times for a pedestrian VRU and at least 5 times for a bicyclist or scooter VRU during Preliminary Acceptance Testing.



Test Number and Title PAT24: VRU approaching Travel Path Function(s) Tested CEAV-030-v01




Procedure

1. Test Conductor performs PAT17 for a VRU approaching the vehicle’s path (rather than already in it)



Evaluation Criteria This will be demonstrated at least 5 times for a pedestrian VRU and at least 5 times for a bicyclist or scooter VRU during Preliminary Acceptance Testing.





Test Number and Title PAT25: Safe Distance to Side of Road Function(s) Tested CEAV-031-v01


Equipment & Setting/Environment Two EZ10 shuttles, static obstacles on open roads Personnel Required Test Manager, Test Conductor

Procedure




maintain a distance from object along the road 5. Test Manager verifies if an object is closer than expected,

verify the shuttle reacts by slowing or stopping

Evaluation Criteria

This will be demonstrated at least 5 times during Preliminary Acceptance Testing. A safe distance is one that does not risk a collision, and in which the other object does not need to take evasive action. This safe distance will be programmed into the vehicle system prior to testing.



Test Number and Title PAT26: Safe Distance Sharing a Lane Function(s) Tested CEAV-032-v01


Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a bicyclist or scooter on open roads


Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle.


3. EZ10 shuttle begins the mission 4. Bicyclist or scooter rides in front of the shuttle 5. Test Manager verifies that the shuttle follows at the same


Evaluation Criteria

This will be demonstrated at least 5 times during Preliminary Acceptance Testing. A safe distance is one that does not risk a collision, and in which the other object does not need to take evasive action. This safe distance will be programmed into the vehicle system prior to testing. Note that it is also acceptable for the vehicle to continue following at a slow speed.





Test Number and Title PAT27: Decreasing Speed due to Uncertainty Function(s) Tested CEAV-033-v01


Equipment & Setting/Environment Two EZ10 shuttles, traffic cone on closed course Personnel Required Test Manager, Test Conductor

Procedure




path or uses a traffic cone to provide an unexpected situation





Test Number and Title PAT28: Detours Function(s) Tested CEAV-034-v01

Test Objective Demonstrate detecting and responding to detours and other temporary changes in traffic patterns.

Equipment & Setting/Environment Two EZ10 shuttles, detour marker on open roads Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor has the vehicle approach traffic cones to

simulate a partial road (lane) closure 5. Test Conductor moves the vehicle around a detour either by

manual control or via pre-programmed alternate 6. Test Conductor has the vehicle approach traffic cones to

simulate a full road closure 7. Test Conductor moves the vehicle around a detour either by

manual control or via pre-programmed alternate

Evaluation Criteria

This will be demonstrated as appropriate along the route. If there are no current changes, it can be simulated with traffic cones. It is met if the vehicle performs the maneuver. A manual override is acceptable for all instances. Note that this test does not include operations on the detour, just the detour process being initiated.





Test Number and Title PAT29: Weather Function(s) Tested CEAV-035-v01

Test Objective Demonstrate operating in rain, fog, and light snow conditions not deemed a weather emergency (when such conditions are available)

Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads in non-standard weather conditions


Procedure


2. Test Conductor allows the shuttle to operate in various conditions, including:

a. Light rain b. Light snow c. Light fog

3. Test Manager verifies the vehicle can operate in these conditions

Evaluation Criteria

During these conditions, considered met if vehicle can continue to operate with similar performance to clear weather conditions. Similar performance includes at normal speed, stopping distance, object detection rate/accuracy, with no unexpected reactions (stopping, swerving, etc.) by the software that is caused by the weather condition. Note that such conditions may not be available during the proposed testing dates. The vehicles will be permitted to proceed, and once these conditions become available, testing will recommence, and the vehicles will be required to pass at that point before they are permitted to operate under such conditions.



Test Number and Title PAT30: Deploy Ramp Function(s) Tested CEAV-036-v01



Procedure



3. Test Conductor pushes the blue button again 4. Test Manager verifies the shuttle first closes the door and


Evaluation Criteria This will be demonstrated at least 5 times during Preliminary Acceptance Testing.





Test Number and Title PAT31: Real-Time Information Function(s) Tested CEAV-037-v01 Test Objective Demonstrate providing real-time information.

Equipment & Setting/Environment Two EZ10 shuttles with a smartphone with the Pivot app installed onboard on open roads

Personnel Required Test Manager, Test Conductor/Data Producer, Data Consumer

Procedure 1. Test Manager uses their smartphones to see real-time

location information on the Pivot app 2. Test Manager views the on-board monitor for current

location and status

Evaluation Criteria Met if able to access accurate vehicle location information from both sources.



Test Number and Title PAT32: Stop and Open Doors at a Transit Stop Function(s) Tested CEAV-039-v01



Procedure



3. EZ10 shuttle begins the mission 4. Test Manager verifies the shuttle doors open when pushing

the door button to board the vehicle 5. Test Manager verifies the shuttle doors open when pushing

the door button to alight the vehicle

Evaluation Criteria This will be demonstrated at least 5 times at each stop location throughout Preliminary Acceptance Testing.



Test Number and Title PAT33: Stop and Open Doors in Emergency Function(s) Tested CEAV-040-v01


Equipment & Setting/Environment Two EZ10 shuttles on closed course Personnel Required Test Manager, Test Conductor

Procedure 1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle



Test Number and Title PAT33: Stop and Open Doors in Emergency 2. Test Conductor assigns the mission to the vehicle to follow

the route 3. EZ10 vehicle begins the mission 4. Test Conductor presses the e-stop button to bring the


the doors 6. Test Manager verifies the doors can be manually opened

Evaluation Criteria This will be demonstrated at least 5 times during Preliminary Acceptance Testing to simulate emergency egress.



Test Number and Title PAT34: Multiple Means of Egress Function(s) Tested CEAV-041-v01 Test Objective Demonstrate that vehicles have multiple means of egress. Equipment & Setting/Environment Two EZ10 shuttles on closed course Personnel Required Test Manager, Test Conductor

Procedure

1. Test Manager verifies that there is an emergency door release button



vehicle




Test Number and Title PAT35: Directional Information Function(s) Tested CEAV-042-v01

Test Objective Demonstrate providing directional (i.e., northbound to St. Stephen’s or southbound to Linden Transit Center) information in audible and visual form to passengers.


Procedure



3. EZ10 shuttle begins the mission 4. Test Manager verifies that audible and visual directions are

given to passengers

Evaluation Criteria The vehicles will switch direction after each run of the route, demonstrating this ability in two different directions.





Test Number and Title PAT36: Cameras Function(s) Tested CEAV-044-v01



Procedure 1. Test Manager visually inspects cameras are installed in the

vehicle 2. Test Manager verifies on EZ Fleet that the cameras can be

monitored




Test Number and Title PAT37: Facility Check Function(s) Tested CEAV-050-v01

Test Objective Show that the vehicles are provided with a safe and secure location for overnight storage, charging, and routine maintenance.

Equipment & Setting/Environment Two EZ10 shuttles in the storage facility Personnel Required Test Manager, Test Conductor

Procedure

1. Test Manager verifies that the facility can fit two or three EZ10 shuttles

2. Test Manager verifies that the facility is secure via a garage door

3. Test Manager verifies that the facility has EV charging capabilities

4. Test Manager verifies that the facility has space to perform light maintenance on the vehicles

Evaluation Criteria The route runs will start and end at the facility, which will be inspected at that time. Met if storage facility is in compliance with the specifications outlined in the RFP, proposal, and contract.



Test Number and Title PAT38: Charging Function(s) Tested CEAV-051-v01

Test Objective Demonstrate the vehicles taking on a charge from each charging port.

Equipment & Setting/Environment Two EZ10 shuttles in the storage facility that is equipped with EV charging



Test Number and Title PAT38: Charging Personnel Required Test Manager, Test Conductor

Procedure 1. Test Manager verifies that EV charging has been installed 2. Test Manager verifies via onboard indicator and EZ Fleet

that when each vehicle is plugged in, it charges

Evaluation Criteria Each vehicle will be plugged in to each of the charging ports, and if it can take a charge it is considered met.



Test Number and Title PAT39: Travel between Route and Facility Function(s) Tested CEAV-052-v01

Test Objective Demonstrate transporting the vehicles between the service route and the facility.

Equipment & Setting/Environment Two EZ10 shuttles on the path to and from the storage facility Personnel Required Test Manager, Test Conductor

Procedure

1. Test Manager verifies that the shuttle can travel between the storage facility and the beginning of the route

2. Test Manager records the travel time 3. Test Manager verifies that the shuttle can travel between

the end of the route and the storage facility 4. Test Manager records the travel time

Evaluation Criteria Testing will begin and end at the facility, so this will be demonstrated prior to the first run and immediately after the last run. Operating in manual mode is acceptable.


8.3. FINAL ACCEPTANCE TESTING During Final Acceptance Testing, each vehicle will first conduct at least 5 runs of a closed course route, primarily to demonstrate certain capabilities as specified throughout this section as being in the closed course environment. Each vehicle will then also conduct at least 20 runs of the service route on open roads over multiple days during different types of conditions, including daylight, nighttime, dawn, dusk, peak traffic and non-peak traffic. Vehicle capabilities will be demonstrated throughout these testing runs. The minimum number of tests for capabilities that do not occur regularly throughout operations, such as deploying the ramp, is specified in the evaluation criteria as necessary.

When the personnel required includes the Test Manager, this may at times be his or her designee.

Table 8-69: Final Acceptance Test 1

Test Number and Title FiAT1: Route Programming Function(s) Tested CEAV-002-v01

Test Objective Ensure the vehicles have been programmed to operate on the route conditions in Columbus.




Test Number and Title FiAT1: Route Programming

Procedure




route

Evaluation Criteria

Met if not a single critical failure (as defined in Section 4.2) is experienced. Requiring a human operator to regularly perform a manual check before telling the vehicle to proceed is acceptable, assuming the vehicle proceeds on its own once it is told the manual check is complete.



Test Number and Title FiAT2: Movement Alert Function(s) Tested CEAV-003-v01

Test Objective Ensure the vehicles have an indicator to alert passengers when the vehicle will begin moving following a designated stop.


Procedure



3. EZ10 shuttle begins the mission 4. Test Manger verifies an audible alert is presented inside





Test Number and Title FiAT3: Incident Notification Function(s) Tested CEAV-004-v01

Test Objective Ensure there are procedures to notify the City in the event of an incident involving the vehicles and/or passengers.

Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads Personnel Required Test Manager, Test Conductor, Stakeholder

Procedure 1. Test Manager and Stakeholder familiarize with SOP that

has been updated after tabletop meeting 2. Test Conductor demonstrates procedure to Test Manager

during the Final Acceptance Testing period



Test Number and Title FiAT3: Incident Notification

Evaluation Criteria Met if there is concurrence by Stakeholders that the approach is complete



Test Number and Title FiAT4: Manual Override Function(s) Tested CEAV-005-v01 Test Objective Ensure a manual override is possible. Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. At a desired point during the mission, Test Conductor takes

over manual control of the vehicle 5. Test Manager verifies that the shuttle is under manual


Evaluation Criteria




Test Number and Title FiAT5: Trained Operators Function(s) Tested CEAV-006-v01

Test Objective Ensure human operators are properly trained to operate the shuttles and interact with passengers.

Equipment & Setting/Environment Documentation of SOP and training procedures Personnel Required Test Manager, Test Conductor Stakeholders

Procedure 1. Test Manager and Stakeholders familiarize with SOP and

Training Procedures 2. Stakeholders engage with Operators during travel 3. Stakeholder witness operation of vehicle by Operators

Evaluation Criteria Two trained operators will demonstrate their ability to operate the vehicles. In addition, EasyMile will share a summary of their training procedures.





Test Number and Title FiAT6: Battery Level Monitoring

Function(s) Tested CEAV-009-v01: Ensure that EasyMile is able to monitor the current battery level at all times.

Test Objective Ensure that in-service vehicle has sufficient charge to operate and will not disrupt operations by stopping along route or in route to charging location, requiring tow or similar activity


Procedure

1. Test Conductor points out indicator to Test Manager 2. Test Manager monitors reading over course of operations to

verify it does indeed reflect battery-level drop 3. Test Manager evaluates that Operator removes vehicle

from service when agreed-upon level is reached

Evaluation Criteria Met if battery performance aligns with specs, and if SOPs include specifications on the level at which a vehicle will be taken out of service.



Test Number and Title FiAT7: Low-Speed Merge Function(s) Tested CEAV-010-v01 Test Objective Demonstrate a low-speed merge. Equipment & Setting/Environment Two EZ10 shuttles, one additional vehicle on open roads Personnel Required Test Manager, Test Conductor

Procedure 1. Test Conductor provides the vehicle with confirmation that it

is safe to proceed at each stop departure. 2. Test Manager verifies that the vehicle proceeds when the


Evaluation Criteria

Met if vehicle performs the maneuver without manual override for 100% of stop departures (operator providing vehicle with confirmation that it is safe to proceed is acceptable, but the operator should use the remote to steer the vehicle unless an obstacle is blocking the path).



Test Number and Title FiAT8: Pulling Over Function(s) Tested CEAV-011-v01 Test Objective Demonstrate pulling over to the side of the road. Equipment & Setting/Environment Two EZ10 shuttles on open roads Personnel Required Test Manager, Test Conductor

Procedure 1. Test Conductor decides the EZ10 shuttle should pull over to

the side of the road in the interest of safety, because of an incident onboard, an unsafe vehicle on the roadway, an



Test Number and Title FiAT8: Pulling Over approaching emergency vehicle, or just to simulate the procedure during testing, etc.

2. Test Conductor engages manual mode 3. Test Manager verifies that the vehicle pulls from the lane of

travel to the side of the road and stops

Evaluation Criteria This will be demonstrated at least 5 times during Final Acceptance Testing.



Test Number and Title FiAT9: Serving a Stop Location Function(s) Tested CEAV-012-v01



Procedure






6. Test Conductor performs FiAT7

Evaluation Criteria This will be demonstrated at each stop arrival. Met if vehicle performs the maneuver without manual override for 100% of stop arrivals. Test can be deferred if an object is blocking the stop location.



Test Number and Title FiAT10: Car Following at an Intersection Function(s) Tested CEAV-013-v01 Test Objective Demonstrate car following when approaching intersections. Equipment & Setting/Environment Two EZ10 shuttles, one additional vehicle on open roads Personnel Required Test Manager, Test Conductor

Procedure






Test Number and Title FiAT10: Car Following at an Intersection 4. Test Conductor has the vehicle lead the shuttle on the route 5. Test Conductor has the vehicle stop at an intersection and

then proceeds when safe 6. Test Manager verifies that the shuttle moves towards the

intersection, stops, and then proceeds when safe Evaluation Criteria This will be demonstrated at each intersection approach.



Test Number and Title FiAT11: Car Following in Stop and Go Traffic Function(s) Tested CEAV-014-v01 Test Objective Demonstrate car following in stop and go traffic conditions.



Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor has the vehicle lead the shuttle on the route 5. Test Conductor has the vehicle brake and go to simulate

stop and go traffic conditions

Evaluation Criteria This will be demonstrated when the vehicle is in mixed traffic (open roads) or by using another test vehicle (closed course) throughout Final Acceptance Testing.



Test Number and Title FiAT12: Navigating Unsignalized Intersections Function(s) Tested CEAV-016-v01 Test Objective Demonstrate how an unsignalized intersection is navigated. Equipment & Setting/Environment Two EZ10 shuttles on open roads Personnel Required Test Manager, Test Conductor

Procedure




the vehicle comes to a stop if it is on the stop-controlled approach.

5. Test Conductor (onboard operator) provides the vehicle with confirmation to proceed when it is safe to do so (i.e., that there are no other vehicles at or approaching the intersection or the EZ10 arrived at the intersection before



Test Number and Title FiAT12: Navigating Unsignalized Intersections any other vehicles and the operator perceives that all other vehicles will stop, will remain stopped, or will not make a conflicting movement)


Evaluation Criteria This will be demonstrated at each intersection approach throughout testing and may include both two-way and four-way stop controlled intersections.



Test Number and Title FiAT13: Left and Right Turns Function(s) Tested CEAV-017-v01 Test Objective Demonstrate a left and a right turn. Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads Personnel Required Test Manager, Test Conductor

Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle that has left and right turn movements



turn and a right turn along the route Evaluation Criteria This will be demonstrated as appropriate along the route.



Test Number and Title FiAT14: Traffic Circle Function(s) Tested CEAV-018-v01

Test Objective Demonstrate entering and emerging from a stop-controlled traffic circle.

Equipment & Setting/Environment Two EZ10 shuttles, one additional vehicle on open roads Personnel Required Test Manager, Test Conductor

Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle that has a traffic circle as part of the route


3. EZ10 shuttle begins the mission 4. Test Manager verifies that the vehicle comes to a stop at

the entrance to the traffic circle 5. At the entrance to the traffic circle, the Test Conductor

(onboard operator) makes appropriate right-of-way decisions and provides the EZ10 with confirmation to proceed when it is the vehicle’s turn

6. EZ10 shuttle proceeds through the traffic circle 7. Test Manager verifies that the vehicle enters the traffic circle

when it is safe and clear of opposing traffic



Test Number and Title FiAT14: Traffic Circle 8. Test Manager verifies that the vehicle exits the traffic circle

at the programmed location Evaluation Criteria This will be demonstrated as appropriate along the route.



Test Number and Title FiAT15: Crossing Intersections Function(s) Tested CEAV-019-v01

Test Objective Demonstrate crossing intersections with traffic speeds up to 35 mph.


Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle that has an intersection as part of the route


3. EZ10 shuttle begins the mission 4. Each time the vehicle reaches an intersection, the Test

Conductor (onboard operator) makes the decision on when it is safe to cross and provides the EZ10 with confirmation to proceed when it is safe

5. Test Manager verifies that the vehicle safety crosses the intersection when no opposing traffic is present

Evaluation Criteria This will be demonstrated as appropriate along the route. Source: City of Columbus


Test Number and Title FiAT16: Merging from a Transit Stop Function(s) Tested CEAV-020-v01

Test Objective Demonstrate appropriate right-of-way decisions when merging from a transit stop.


Procedure

1. Test Conductor loads a pre-programmed route onto an EZ10 shuttle with at least four stops


3. EZ10 shuttle begins the mission 4. At each stop departure, the Test Conductor (onboard

operator) provides the shuttle with confirmation to proceed after stop operations (boarding and alighting) have been completed and once the roadway is clear

5. Test Manager verifies that the vehicle proceeds when the roadway is clear, and that it waits if it is not clear



Test Number and Title FiAT16: Merging from a Transit Stop

Evaluation Criteria This will be demonstrated at each stop departure. The appropriate maneuver is waiting until it is clear to pull out.



Test Number and Title FiAT17: Intersection Right-of-Way Function(s) Tested CEAV-021-v01



Procedure




the vehicle comes to a stop if it is on the stop-controlled approach



7. EZ10 shuttle proceeds through the intersection if no vehicles are approaching the intersection

Evaluation Criteria

This will be demonstrated at least 5 times during Final Acceptance Testing. The appropriate maneuver is for the vehicle to proceed when it’s the vehicle’s turn. If all approaches are stop-controlled, it will proceed based on order of arrival. If there is no stop-control on its approach, it will continue moving through the intersection.



Test Number and Title FiAT18: VRU Right-of-Way Function(s) Tested CEAV-022-v01


Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a VRU (pedestrian, bicyclist, scooter) on closed course and open roads


Procedure






Test Number and Title FiAT18: VRU Right-of-Way 4. VRU stands in the road at a point along the route or



6. VRU walks or rides along the programmed route 7. EZ10 shuttle follows at a safe speed 8. EZ10 passes the VRU when it is safe to do so (by engaging

manual mode) 9. Once the VRU is out of the path, verify normal operation

resumes

Evaluation Criteria

This will be demonstrated at least 10 times for a pedestrian VRU and at least 10 times for a bicyclist or scooter VRU during Final Acceptance Testing. The appropriate maneuvers include yielding for a VRU crossing the vehicle’s path, following a slower moving VRU, and overtaking a stopped VRU (using manual mode). See FiAT27 and FiAT28 for variants and additional tests.



Test Number and Title FiAT19: Encroaching Oncoming Vehicles Function(s) Tested CEAV-023-v01




Procedure




assigned path 5. Test Manager verifies the shuttle reacts appropriately, either

slowing or stopping based on the closeness of the approaching vehicle

Evaluation Criteria This will be demonstrated at least 5 times during Final Acceptance Testing. The appropriate maneuvers include slowing down, stopping, and possibly prompting the onboard operator for guidance or to take control.



Test Number and Title FiAT20: Detect Stopped Vehicle Function(s) Tested CEAV-024-v01



Test Number and Title FiAT20: Detect Stopped Vehicle Test Objective Demonstrate detecting stopped vehicles in the path.



Procedure



3. EZ10 shuttle begins the mission 4. The additional vehicle is parked in the road in EZ10’s

programmed path 5. Test Manager verifies the shuttle slows to a stop as it


Evaluation Criteria This will be demonstrated at least 5 times during Final Acceptance Testing. The appropriate maneuver is for the vehicle to stop and prompt for additional guidance.



Test Number and Title FiAT21: Pass Stopped Vehicle Function(s) Tested CEAV-025-v01 Test Objective Demonstrate passing a stopped vehicle when safe



Procedure 1. Test Conductor performs FiAT22. 2. Test Conductor engages manual mode and passes the


Evaluation Criteria This will be demonstrated at least 5 times during Final Acceptance Testing. The appropriate maneuver is for the vehicle to wait until the opposing direction is clear and then pass.



Test Number and Title FiAT22: Static Obstacles Function(s) Tested CEAV-026-v01 Test Objective Demonstrate detecting and responding to static obstacles. Equipment & Setting/Environment Two EZ10 shuttles on closed course Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor loads the sensor visualization 5. Test Conductor presents objects



Test Number and Title FiAT22: Static Obstacles 6. Test Manager verifies that static objects are detected by the

sensors 7. Test Manager verifies that the shuttle proceeds only when

clear

Evaluation Criteria

This will be demonstrated at least 5 times during Final Acceptance Testing for each of the following objects:

• Parked cars • Trees • Curbs • Buildings • Garbage cans • Piles of leaves (when available, similar to weather testing) • Piles of snow (when available, similar to weather testing)



Test Number and Title FiAT23: Moving Obstacles Function(s) Tested CEAV-027-v01 Test Objective Demonstrate detecting and responding to moving obstacles. Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a VRU on closed course Personnel Required Test Manager, Test Conductor

Procedure





clear

Evaluation Criteria

This will be demonstrated at least 5 times during Final Acceptance Testing for each of the following objects:




Test Number and Title FiAT24: Emergency Vehicle Interaction Function(s) Tested CEAV-028-v01

Test Objective Ensure there are procedures to ensure safe and appropriate interactions with emergency vehicles.



Test Number and Title FiAT24: Emergency Vehicle Interaction Equipment & Setting/Environment Staff and stakeholders at tabletop meeting Personnel Required Test Manager, Test Conductor, Stakeholder

Procedure 1. Test Manager and Stakeholder familiarize with SOP that

has been updated after tabletop meeting 2. Test Conductor demonstrates procedure to Test Manager

during the Final Acceptance Testing period

Evaluation Criteria Met if there is concurrence by Stakeholders that the approach is complete. The appropriate maneuver is for the vehicle to stop and prompt the onboard operator for additional guidance.



Test Number and Title FiAT25: VRU in Travel Path Function(s) Tested CEAV-029-v01




Procedure

1. Test Conductor performs FiAT20 for a VRU in the vehicle’s path



Evaluation Criteria This will be demonstrated at least 5 times for a pedestrian VRU and at least 5 times for a bicyclist or scooter VRU during Final Acceptance Testing.



Test Number and Title FiAT26: VRU approaching Travel Path Function(s) Tested CEAV-030-v01




Procedure

1. Test Conductor performs FiAT20 for a VRU approaching the vehicle’s path (rather than already in it)





Test Number and Title FiAT26: VRU approaching Travel Path

Evaluation Criteria This will be demonstrated at least 5 times for a pedestrian VRU and at least 5 times for a bicyclist or scooter VRU during Final Acceptance Testing.



Test Number and Title FiAT27: Safe Distance to Side of Road Function(s) Tested CEAV-031-v01


Equipment & Setting/Environment Two EZ10 shuttles, static obstacles on open roads Personnel Required Test Manager, Test Conductor

Procedure




maintain a distance from object along the road 5. Test Manager verifies if an object is closer than expect, the

shuttle reacts by slowing or stopping.

Evaluation Criteria

This will be demonstrated at least 5 times during Final Acceptance Testing. A safe distance is one that does not risk a collision, and in which the other object does not need to take evasive action. This safe distance will be programmed into the vehicle system prior to testing.



Test Number and Title FiAT28: Safe Distance Sharing a Lane Function(s) Tested CEAV-032-v01


Equipment & Setting/Environment Two EZ10 shuttles, staff acting as a bicyclist or scooter on open roads


Procedure



3. EZ10 shuttle begins the mission 4. Bicyclist or scooter rides in front of the shuttle 5. Test Manager verifies that the shuttle follows at the same




Test Number and Title FiAT28: Safe Distance Sharing a Lane

Evaluation Criteria

This will be demonstrated at least 5 times during Final Acceptance Testing. A safe distance is one that does not risk a collision, and in which the other object does not need to take evasive action. This safe distance will be programmed into the vehicle system prior to testing. Note that it is also acceptable for the vehicle to continue following at a slow speed.



Test Number and Title FiAT29: Decreasing Speed due to Uncertainty Function(s) Tested CEAV-033-v01


Equipment & Setting/Environment Two EZ10 shuttles, traffic cone on closed course Personnel Required Test Manager, Test Conductor

Procedure




path or uses a retroreflective traffic cone to provide an unexpected situation





Test Number and Title FiAT30: Detours Function(s) Tested CEAV-034-v01

Test Objective Demonstrate detecting and responding to detours and other temporary changes in traffic patterns.

Equipment & Setting/Environment Two EZ10 shuttles, detour marker on open roads Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission 4. Test Conductor has the vehicle approach traffic cones to

simulate a partial road (lane) closure 5. Test Conductor moves the vehicle around a detour either by




Test Number and Title FiAT30: Detours 6. Test Conductor has the vehicle approach traffic cones to

simulate a full road closure 7. Test Conductor moves the vehicle around a detour either by


Evaluation Criteria

This will be demonstrated as appropriate along the route. If there are no current changes, it can be simulated with traffic cones. It is met if the vehicle performs the maneuver. A manual override is acceptable for all instances. Note that this test does not include operations on the detour, just the detour process being initiated.



Test Number and Title FiAT31: Weather Function(s) Tested CEAV-035-v01

Test Objective Demonstrate operating in rail, fog, and light snow conditions not deemed a weather emergency (when such conditions are available)

Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads in non-standard weather conditions


Procedure


2. Test Conductor allows the shuttle to operate in various conditions, including:

a. Light rain b. Light snow c. Light fog

3. Test Manager verifies the vehicle can operate in these conditions

Evaluation Criteria

During these conditions, considered met if vehicle can continue to operate with similar performance to clear weather conditions. Similar performance includes at normal speed, stopping distance, object detection rate/accuracy, with no unexpected reactions (stopping, swerving, etc.) by the software that is caused by the weather condition. Note that such conditions may not be available during the proposed testing dates. The vehicles will be permitted to proceed, and once these conditions become available, testing will recommence, and the vehicles will be required to pass at that point before they are permitted to operate under such conditions.



Test Number and Title FiAT32: Deploy Ramp Function(s) Tested CEAV-036-v01


Equipment & Setting/Environment Two EZ10 shuttles on closed course and open roads



Test Number and Title FiAT32: Deploy Ramp Personnel Required Test Manager, Test Conductor

Procedure



3. Test Conductor pushes the blue button again 4. Test Manager verifies the shuttle first closes the door and


Evaluation Criteria This will be demonstrated at least 5 times during Final Acceptance Testing, at least once at each stop location.



Test Number and Title FiAT33: Real-Time Information Function(s) Tested CEAV-037-v01 Test Objective Demonstrate providing real-time information.

Equipment & Setting/Environment Two EZ10 shuttles with a smartphone with the Pivot app installed onboard on open roads

Personnel Required Test Manager, Test Conductor/Data Producer, Data Consumer

Procedure 1. Test Manager uses their smartphones to see real-time

location information on the Pivot app 2. Test Manager views the on-board monitor for current

location and status

Evaluation Criteria Met if able to access accurate vehicle location information from both sources.



Test Number and Title FiAT34: Real-Time Passenger Count Information Function(s) Tested CEAV-038-v01

Test Objective Demonstrate employing technology to count passenger boardings and alightings, with both location and time attributes, and report these on a daily basis through an API to the Operating System.

Equipment & Setting/Environment Two EZ10 shuttles, EZ Fleet, Operating System access Personnel Required Test Manager, Test Conductor/Data Producer, Data Consumer

Procedure

1. Based on the testing ridership, Test Manager verifies that passenger counts were conducted

2. Test Manager verifies that the API is reporting the counts 3. Test Manager verifies this data is updated every twice per

second in the API Evaluation Criteria Met if 100% success rate (all report requests completed).





Test Number and Title FiAT35: Stop and Open Doors at a Transit Stop Function(s) Tested CEAV-039-v01



Procedure



3. EZ10 shuttle begins the mission 4. Test Manager verifies the shuttle doors open when pushing

the door button to board the vehicle 5. Test Manager verifies the shuttle doors open when pushing

the door button to alight the vehicle

Evaluation Criteria This will be demonstrated at least 5 times at each stop location throughout Final Acceptance Testing.



Test Number and Title FiAT36: Stop and Open Doors in Emergency Function(s) Tested CEAV-040-v01


Equipment & Setting/Environment Two EZ10 shuttles on closed course Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 vehicle begins the mission 4. Test Conductor presses the e-stop button to bring the


the doors 6. Test Manager verifies the doors can be manually opened.

Evaluation Criteria This will be demonstrated 5 times during Final Acceptance Testing to simulate emergency egress.



Test Number and Title FiAT37: Multiple Means of Egress Function(s) Tested CEAV-041-v01



Test Number and Title FiAT37: Multiple Means of Egress Test Objective Demonstrate that vehicles have multiple means of egress. Equipment & Setting/Environment Two EZ10 shuttles on closed course Personnel Required Test Manager, Test Conductor

Procedure

1. Test Manager verifies that there is an emergency door release button



vehicle




Test Number and Title FiAT38: Directional Information Function(s) Tested CEAV-042-v01

Test Objective Demonstrate providing directional (i.e., northbound to St. Stephen’s or southbound to Linden Transit Center) information in audible and visual form to passengers.


Procedure



3. EZ10 shuttle begins the mission 4. Test Manager verifies that audible and visual directions are

given to passengers

Evaluation Criteria Test Manager will check after each turnaround point and at least once within each directional run that the sources are showing the correct direction.



Test Number and Title FiAT39: Onboard Wi-Fi Function(s) Tested CEAV-043-v01 Test Objective Demonstrate providing free Wi-Fi to onboard passengers. Equipment & Setting/Environment Two EZ10 shuttles (that are Wi-Fi enabled), Speedtest app Personnel Required Test Manager, Test Conductor

Procedure

1. Test Manager boards the shuttle 2. Test Manager uses a smartphone to connect to the shuttle

Wi-Fi 3. Test Manager verifies there is no cost to connect 4. Test Manager opens Speedtest app and begins test to

verify connection is active



Test Number and Title FiAT39: Onboard Wi-Fi

Evaluation Criteria Three passengers will attempt to use the onboard Wi-Fi for each vehicle. Met if all three are able to access and use the Wi-Fi.



Test Number and Title FiAT40: Cameras Function(s) Tested CEAV-044-v01



Procedure 1. Test Manager visually inspects cameras are installed in the

vehicle 2. Test Manager verifies on EZ Fleet that the cameras can be

monitored




Test Number and Title FiAT41: Weather Monitoring Function(s) Tested CEAV-045-v01

Test Objective Demonstrate how local weather patterns will be monitored to maintain awareness of any approaching severe weather event or other conditions that may impact vehicle operations.

Equipment & Setting/Environment Computer interface that is being monitored by a staff center Personnel Required Test Manager, Test Conductor

Procedure 1. Test Manager verifies the staff has a means of monitoring

the local weather conditions via web portal or via television 2. Test Manager verifies this process is documented in the

SOP and demonstrated at the tabletop meeting

Evaluation Criteria Testing team will visit monitoring location on two occasions and analyze weather monitoring operations. Procedure will also be documented in SOPs. Met if system is considered satisfactory to reliably take the vehicles out of service when necessary.



Test Number and Title FiAT42: Headway Function(s) Tested CEAV-046-v01

Test Objective Demonstrate the ability to meet a minimum headway of twelve minutes during hours in which service is provided.

Equipment & Setting/Environment Two EZ10 shuttles on open roads



Test Number and Title FiAT42: Headway Personnel Required Test Manager, Test Conductor

Procedure



3. EZ10 shuttle begins the mission with both EZ10 shuttles 4. Test Manager monitors the progress and calculate the

headways between stops

Evaluation Criteria Route run times will be monitored during each run for each vehicle to ensure this is feasible with the fleet size. Met if average route run time is under 24 minutes.



Test Number and Title FiAT43: Real-Time Data Function(s) Tested CEAV-047-v01

Test Objective Demonstrate collecting required data and sending this data to the Smart Columbus Operating System via an API in real-time or near real-time.


Procedure

1. Test Manager verifies the API is active in the EZ Fleet system

2. Test Manager verifies the Operating System is receiving data from the API

3. Test Manager verifies the data is near real-time at twice per second

Evaluation Criteria Met if data is transmitted successfully at the required latency. Source: City of Columbus


Test Number and Title FiAT44: Daily Reported Data Function(s) Tested CEAV-048-v01

Test Objective Demonstrate collecting required data and sending this data to the Smart Columbus Operating System via an API on a daily basis.


Procedure 1. Test Conductor performs FiAT45 2. Test Manager verifies the data can be visualized in a daily

format Evaluation Criteria Met if data is transmitted successfully.





Test Number and Title FiAT45: Weekly Reported Data Function(s) Tested CEAV-049-v01

Test Objective Demonstrate collecting required data and sending this data to the Smart Columbus Operating System via an API on a weekly basis.


Procedure 1. Test Conductor performs FiAT45 2. Test Manager verifies the data can be visualized in a

weekly format. Evaluation Criteria Met if data is transmitted successfully.



Test Number and Title FiAT46: Charging Function(s) Tested CEAV-051-v01

Test Objective Demonstrate the vehicles taking on a charge from each charging port.

Equipment & Setting/Environment Two EZ10 shuttles in the storage facility that is equipped with EV charging


Procedure 1. Test Manager verifies that EV charging has been installed 2. Test Manager verifies via onboard indicator and EZ Fleet

that when each vehicle is plugged in, it charges

Evaluation Criteria Each vehicle will be plugged in to each of the charging ports, and if it can take a charge it is considered met.



Test Number and Title FiAT47: Travel between Route and Facility Function(s) Tested CEAV-052-v01

Test Objective Demonstrate transporting the vehicles between the service route and the facility.

Equipment & Setting/Environment Two EZ10 shuttles on the path to and from the storage facility Personnel Required Test Manager, Test Conductor

Procedure

1. Test Manager verifies that the shuttle can travel between the storage facility and the beginning of the route.

2. Test Manager records the travel time. 3. Test Manager verifies that the shuttle can travel between

the end of the route and the storage facility. 4. Test Manager records the travel time.

Evaluation Criteria Testing will begin and end at the facility, so this will be demonstrated prior to the first run and immediately after the last run. Operating in manual mode is acceptable.





Appendix A. Terminology and Conventions

A.1 NUMBERING CONVENTION Each testing element contains a unique identifier for traceability and configuration management. Test cases and scenarios for all projects in the Smart Columbus program will follow the same convention, each representing an identifiable attribute of the traced metric. The convention is as follows:

Figure 8-1: Numbering Convention Source: City of Columbus

Table 8-116: Numbering Convention Definitions

Octet Description Data Type, Casing

Number of Characters or Digits

a. Project Abbreviation

The designated Smart Columbus project acronym (i.e., CEAV)

String, upper case

Variable

b. Test Type Code Note that Test Type Codes are not currently being used for this project.

XYZ: Test Case for the XYZ XYZ: Test Case for the XYZ • XYZ: Test Case for the XYZ XYZ: Test Case for the XYZ TES: Test Scenario

String, upper case

3

c. Test Number An integer incrementing by one, indicating the number of requirements established.

Integer 3

d. “v” Static Character

Static letter “v” represents the version for the particular test objective and procedure.

Character 1

e. Version Number An integer incrementing by one, indicating the number of revisions made to the test element being traced.

Integer 2


Project Abbreviation

Test Type Code

Test Number

Version Number

“V” Static Character

01 02 03

(a) (b) (c) (d) (e)

Appendix A. Terminology and Conventions


An example of a test case for the ability of the CEAV to navigate unsignalized intersections would be CEAV-016-v01.

1. “CEAV” is the project abbreviation.

2. “016” is the test type code coupled with the three-digit test number.

“v01” is the static “v” coupled with the two-digit version number.


Appendix B. Acronyms and Definitions

Table 8-117 contains project specific acronyms used throughout this document.

Table 8-117: Acronym List

Abbreviation/Acronym Definition

ADA Americans with Disabilities Act

API Application Programming Interface

BRT Bus Rapid Transit

CEAV Connected Electric Autonomous Vehicle

CMAX COTA’s Bus Rapid Transit (BRT) Service

ConOps Concept of Operations

COTA Central Ohio Transit Authority

CV Connected Vehicle

DF Desirable Function

DSRC Dedicated Short Range Communications

EF Essential Function

FHWA Federal Highway Administration

FMLM First Mile/Last Mile

GPS Global Positioning System

ITS Intelligent Transportation System

MAP Intersection Geometry Message

ODOT Ohio Department of Transportation

OpCon Operational Concept

OS Operating System

OSU Ohio State University

RFP Request for Proposals

RSU Roadside Unit (DSRC)

SC Smart Columbus

SMH Smart Mobility Hub

SOP Standard Operating Procedures

SPaT Signal Phase and Timing

Appendix B. Acronyms and Definitions


SSL Secure Sockets Layer

USDOT U.S. Department of Transportation



Appendix C. Glossary

Table 8-118 contains project specific terms used throughout this document.

Table 8-118: Glossary

Term Definition

Commercial Off-the-Shelf System (COTS)

Software or hardware product that are ready-made and available for sale to the public. Also referred to as a turnkey system.

Normal Operations When a signalized intersection is cycling through its preplanned phases correctly, servicing all approaches including pedestrian phases.

Procurement The act of obtaining or acquiring goods, services or works, from a competitive bidding process.

Real-Time Data Information that is delivered immediately after collection.



Appendix D. Requirements Summary

The following table provides a list of the requirements outlined in the RFP and subsequent procurement documentation provided by EasyMile. Note that some of these requirements are contractual requirements that will not be tested (such as the vehicle delivery date), but they are still included here for reference.

Table 8-119: Requirements

ID Requirement Source CEAV-OPS-

001-v01 • Vehicles shall be delivered by September 9, 2019, to allow

testing to commence in mid-September 2019 and passenger service to commence in November 2019. Passenger service is expected to be provided for one full year, November 2019 to November 2020. o Up to a week prior to the launch of public passenger service, a

soft launch of full service shall commence to facilitate Partner Preview and Media Days.

RFP (pg. 19), contract (A.3)

CEAV-OPS-002-v01

• The Offeror shall provide service between the hours of 6 AM and 8 PM, seven days a week. Ridership shall be monitored by time-of-day and day-of-week, and operating hours may be adjusted to better accommodate demand, considering vehicle capabilities. For example, weekend service to St. Stephen’s may not be required and the decision may be made to terminate the route at the Douglas Community Recreation Center on weekends. o Service may be suspended or reduced on Christmas Day,

though it is desired that service be continued on other holidays consistent with the availability of services offered along the route, including St. Stephen’s Community House, Douglas Community Recreation Center, and the Linden Transit Center.

o Daily hours of operation shall be communicated to passengers in line with the approved Marketing and Education Plan and Communications and Engagement Plan deliverables. Any proposed changes to hours of operation shall be communicated to Smart Columbus at least two weeks in advance.


CEAV-OPS-003-v01

• The Offeror shall meet a minimum headway of twelve (12) minutes during hours in which service is provided. This is anticipated to be achievable on the proposed route with a minimum of two (2) vehicles in service at any time. As with operating hours, desired minimum headway may be modified during certain time periods depending on ridership but shall remain within the capabilities of the Offeror’s originally proposed vehicle fleet size. Stop departure times shall be scheduled to complement nearby COTA services.


CEAV-OPS-004-v01

• The Offeror is responsible for identifying and obtaining a secure location for overnight storage, performance of maintenance work,




ID Requirement Source and charging of the vehicles within a suitable distance of the proposed routes. o The Offeror is responsible for transporting the vehicles to/from

the route and this storage facility, as well as to a nearby charging station, if applicable. Operating the vehicles in automated or manual mode for this task is acceptable, and regularly towing the vehicles is not desirable.

o This storage facility shall accommodate access that supports the full hours of operation defined herein.

CEAV-OPS-005-v01

• The Offeror shall be responsible for keeping the vehicles charged.


CEAV-OPS-006-v01

• The Offeror shall train onboard operators who will always be on board a vehicle when it is operating.


CEAV-OPS-007-v01

• Onboard operations shall have received training to: o Assist and interact with passengers, including providing

mobility assistance during passenger boarding and alighting, as necessary

o Provide accurate basic information about the vehicle, the purpose of the route, and the Smart Columbus program

o Receive and record passenger feedback o Operate a ramp, door, and/or charging station, if not

automated o Perform road testing of a vehicle o Have a working knowledge of vehicle equipment o Perform clean-up, including bodily fluid o Intervene in vehicle operations, if necessary

RFP (pg. 19-20), contract (A.6)

CEAV-OPS-008-v01

• Operators shall be employees, contractors, or agents of the Offeror and also maintain: o Defensive driving certification o First Aid training o A valid driver’s license that is recognized by the State of Ohio

(note: if a City permit is required, an Ohio driver’s license may be required)

o No more than two traffic violations or preventable accidents in the last three years


CEAV-OPS-009-v01

• It is the responsibility of the Offeror to attain all licenses and permits to deploy the vehicles and operators. It is anticipated that the Offeror will need to apply for the following: 1. State of Ohio Autonomous Vehicle Operations Permit 2. Bureau of Motor Vehicles (BMV) vehicle registration and

licensing 3. Either: City of Columbus Micro Transit Operator – Vehicle for Hire

Owner and Vehicle for Hire Driver; or Public Utilities Commission of Ohio (PUCO) Commercial

Motor Vehicle 4. City of Columbus (or relevant jurisdiction/entity) attachment

permit if pole mounted equipment is required for operation The City does not guarantee that this list is exhaustive of all licenses and permits but is provided to aid the Offeror in developing an approach to deploy and developing the price for its response.




ID Requirement Source CEAV-OPS-

010-v01 • Any fleet management system(s) shall be open architecture to

allow for potential integration with the Smart Columbus Operating System.


CEAV-OPS-011-v01

• The Offeror shall be responsible for developing Standard Operating Procedures for the vehicles and operations staff.

RFP (pg. 20), contract (A.11.2)

CEAV-OPS-012-v01

• The Offeror shall maintain the latest vehicle software update at all times at no additional cost to Smart Columbus.


CEAV-OPS-013-v01

• The Offeror shall monitor information on the vehicles battery level, ensuring the vehicles are sufficiently charged or taken out of service early under abnormal conditions after servicing all passengers who are already on board.


CEAV-OPS-014-v01

• The Offeror shall monitor local weather patterns and be aware of any approaching severe weather event or other conditions that may impact vehicle operations. When a Level 1 Emergency or above (as defined by Franklin County) is announced, operations will be suspended. The Offeror may also define an inclement weather threshold (such as snow depth on the roadway, visibility, or other physical limitations) at which it would suspend or limit operations or shift to manual mode and define its response procedure in its Project Plan. The Offeror shall agree to notify Smart Columbus in the event this inclement weather threshold is met.


CEAV-OPS-015-v01

• The Offeror shall work in earnest with OSU to identify and/or support research opportunities.


CEAV-OPS-016-v01

• The Offeror shall commit the capacity and resources necessary to market and co-promote the pilot and build local stakeholder relationships along the route to increase user adoption.


CEAV-OPS-017-v01

• The Offeror shall actively engage the community in the operation of the vehicle and monitor their feedback. This can be accomplished through surveys of both riders and non-riders. Surveys may be distributed on board the vehicle, via either an onboard tablet or information that may allow passengers to complete a survey through their personal device. Survey questions will be provided by the City in coordination with OSU. It is expected that the survey will be similar in nature to the one being deployed with the Smart Circuit around the Scioto Mile.


CEAV-OPS-018-v01

• The Offeror shall immediately notify Smart Columbus of any crashes or incidents related to transporting customers.


CEAV-OPS-019-v01

• The Offeror shall designate an employee as a liaison with Smart Columbus and shall provide Smart Columbus with contact information, including a mobile phone number, where the liaison can be reached.


CEAV-OPS-020-v01

• The Offeror shall provide a call-in number for Smart Columbus to contact in case of emergency, with an immediate response available during all operating hours.


CEAV-OPS-021-v01

• During the performance of any contract resulting from this RFP, the Offeror, for itself, its assignees, and successors in interest agrees to comply with the following non-discrimination statutes and authorities; including but not limited to: o Title VI of the Civil Rights Act of 1964 (42 U.S.C. § 2000d et

seq., 78 stat. 252) (prohibits discrimination on the basis of race, color, or national origin, and also requires programs

RFP (pg. 21), contract (Section

1)



ID Requirement Source receiving Federal funding to provide equal access to all regardless of income level or access to technology such as smartphones); and 49 CFR Part 21

o The Uniform Relocation Assistance and Real Property Acquisition Policies Act of 1970 (42 U.S.C. § 4601) (prohibits unfair treatment of persons displaced or whose property has been acquired because of Federal or Federal-Aid programs and projects)

o Federal-Aid Highway Act of 1973 (23 U.S.C. § 324 et seq.) (prohibits discrimination on the basis of sex)

o Section 504 of the Rehabilitation Act of 1973 (29 U.S.C. § 794 et seq.), as amended (prohibits discrimination on the basis of disability) and 49 CFR Part 27

o The Age Discrimination Act of 1975, as amended (42 U.S.C. § 6101 et seq.) (prohibits discrimination on the basis of age)

o The Civil Rights Restoration Act of 1987 (PL 100-209) (broadened the scope, coverage, and applicability of Title VI of the Civil Rights Act of 1964, the Age Discrimination Act of 1975, and Section 504 of the Rehabilitation Act of 1973, by expanding the definition of the terms “programs or activities” to include all of the programs or activities of Federal-Aid recipients, sub-recipients, and contractors, whether such programs or activities are Federally funded or not)

o Titles II and III of the Americans with Disabilities Act (42 U.S.C. §§ 12131-12189), as implemented by Department of Transportation regulations at 49 CFR parts 37 and 38 (prohibits discrimination on the basis of disability in the operation of public entities, public and private transportation systems, places of public accommodation, and certain testing entities, including for passengers with mobility access issues as well as those with visual, hearing, cognitive, and other impairments)

o Executive Order 12898, Federal Actions to Address Environmental Justice in Minority Populations and Low-Income Populations (ensures non-discrimination against minority populations by discouraging programs, policies, and activities with disproportionately high and adverse human health or environmental effects on minority and low-income populations)

o Executive Order 13166, Improving Access to Services for People with Limited English Proficiency, and resulting agency guidance, national origin discrimination includes discrimination because of limited English proficiency (LEP). To ensure compliance with Title VI, Offerors must take reasonable steps to ensure that LEP persons have meaningful access to their programs (70 Fed. Reg. at 74087 to 74100)

o Title VIII of the Civil Rights Act of 1968 (Fair Housing Act), as amended (prohibits discrimination in the sale, rental, and financing of dwellings on the basis of race, color, religion, sex, national origin, disability, or familial status, such as presence of child under the age of 18 and pregnant women)

o Title IX of the Education Amendments Act of 1972, as amended (20 U.S.C. 1681 et seq.) (prohibits discrimination on the basis of sex in education programs or activities)



ID Requirement Source o Uniformed Services Employment and Reemployment Rights

Act (USERRA) (38 U.S.C. 4301-4333) (prohibits discrimination on the basis of present, past or future military service)

o Genetic Information Nondiscrimination Act (GINA) (29 CFR Part 1635, 42 U.S.C. 2000ff)

CEAV-OPS-022-v01

• Prior to the beginning of passenger service, the Selected Offeror shall meet with City of Columbus emergency services (fire, police, ambulance, etc.) to provide background and operating information on the vehicles. Participation in a tabletop exercise detailing various response scenarios that test the thoroughness and accuracy of the Standard Operating Procedures is required.


CEAV-VEH-001-v01

• Vehicles must be model/manufacturer year 2016 or newer, and free of any major dents, scratches, or other damage that may present a potential safety hazard or be cosmetically unappealing. Tires must be in good condition.


CEAV-VEH-002-v01

• The Offeror shall include responses for the vehicle to the USDOT National Highway Traffic Safety Administration (NHTSA) 12-point safety assessment, as well as whether the assessment has been submitted to NHTSA and, if not, whether there are any plans to do so.

RFP (pg. 22), proposal

CEAV-VEH-003-v01

• The vehicles shall comply with all applicable FMVSS or have approval to operate under an exemption to the FMVSS. If not compliant, describe how the items not in compliance are directly related to the full automation capability with no driver.


CEAV-VEH-004-v01

• Each vehicle shall have a capacity of at least 4 passengers excluding the operator. The vehicle shall also have space for passengers to store small amounts of luggage, such as grocery bags and strollers.


CEAV-VEH-005-v01

• The vehicles should implement Dedicated Short-Range Communications (DSRC) to communicate with roadside equipment to receive signal phase and timing (SPaT) and MAP data. While the route does not cross a signalized intersection, the vehicle will encroach upon Cleveland Avenue and 11th Avenue which will be a DSRC-equipped intersection. The project team is interested in evaluating the information received by the vehicle even though it will not be required to process the information to navigate the route. o If the vehicles do not have this capability, at a minimum the

Offeror shall agree to install an onboard unit (OBU) supplied by Smart Columbus on each vehicle. This will allow the vehicles to communicate information on their location and share basic safety messages (BSM) with other connected vehicles in the Smart Columbus Connected Vehicle Environment and with vehicles equipped by OSU.5 The shuttle vehicles are not required to use the information they receive through the OBUs to inform decisions, though this is desirable. The Selected Offeror shall provide a means from which data from these OBUs will be collected.

o A commercial Security Credential Management System (SCMS) will be provided by the City of Columbus for the vehicles to enroll in.


As explained in Chapter 2, this

capability will be tested as part of the separate CV

Environment project and not

the CEAV project.



ID Requirement Source CEAV-VEH-

006-v01 • The vehicles shall be equipped with cameras capable of viewing

and recording the entirety of the passenger compartment. Further, the vehicles shall be equipped with cameras capable of capturing a 360 degree view external to the vehicle. o Video footage shall be collected and stored by the Selected

Offeror and be accessible to Smart Columbus and/or the Columbus Police Department if requested.

o Audio footage of the passenger compartment may also be collected.


CEAV-VEH-007-v01

• The Offeror shall agree to allow the vehicles to be wrapped or otherwise branded consistent with the intent of the deployment. Branding may include the Offeror’s logo if desired alongside other graphics and sponsor brands. The Offeror shall provide limitations on placement of branding, so as to not occlude vital system functions, as part of its proposal. The final design will be coordinated with Smart Columbus.


CEAV-VEH-008-v01

• Vehicles shall be equipped with technology to count passenger boardings and alightings, with both location and time attributes, and this data shall be reported on a daily basis through an API to the Operating System.


CEAV-VEH-009-v01

• The vehicles shall stop and open doors at designated locations to allow passengers to board and alight. The vehicles shall not park in a spot blocking access to a fire hydrant or crosswalk. The vehicles shall also stop and open doors if they have detected that there is an issue on board, through passenger input and/or secure override. The vehicles should also have multiple secure means of egress, in the event the primary exit is blocked and/or power failure occurs.


CEAV-VEH-010-v01

• The vehicles shall be capable of providing directional (i.e., northbound to St. Stephen’s or southbound to Linden Transit Center) information in audible and visual form to passengers on both the inside and the outside of the vehicle.

RFP (pg. 23)

CEAV-VEH-011-v01

• The Offeror shall identify the ability of its vehicles to operate the following operating functions in automated mode: o Performing a low-speed merge

RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-012-v01

o Pulling over to the side of the road RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-013-v01

o Moving out of the travel lane and stopping in order to service stop locations


CEAV-VEH-014-v01

o Performing car following when approaching intersections RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-015-v01

o Performing car following in stop and go traffic conditions by maintaining a safe distance behind the vehicle in front of them and determining when to proceed based on that vehicle's behavior


CEAV-VEH-016-v01

o Navigating signalized (i.e., with the use of DSRC) intersections


CEAV-VEH-017-v01

o Navigating unsignalized intersections RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-018-v01

o Performing left and right turns RFP (pg. 23, 27), contract (A.12)




019-v01 o Entering and emerging from a stop-controlled traffic circle RFP (pg. 23, 27),

contract (A.12) CEAV-VEH-

020-v01 o Crossing intersections with traffic speed limits up to 35 mph

(and at times exceeding the posted speed limit) RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-021-v01

o Changing lanes (both left and right lane change) RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-022-v01

o Making appropriate right-of-way decisions when merging from a transit stop


CEAV-VEH-023-v01

o Making appropriate right-of-way decisions at intersections RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-024-v01

o Making appropriate right-of-way decisions when interacting with vulnerable road users


CEAV-VEH-025-v01

o Detecting and responding to encroaching oncoming vehicles RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-026-v01

o Detecting stopped vehicles in their path RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-027-v01

o Passing stopped vehicles when safe RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-028-v01

o Detecting and responding to static obstacles in their path RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-029-v01

o Detecting and responding to moving obstacles in their path RFP (pg. 23, 27), contract (A.12)

CEAV-VEH-030-v01

o Detecting emergency vehicles, and when their sirens are on, and yielding appropriately


CEAV-VEH-031-v01

o Detecting that they are being asked by law enforcement to move to the side of the road, and responding accordingly


CEAV-VEH-032-v01

o Detecting and responding to vulnerable road users, such as pedestrians, cyclists, and scooters, in the vehicle’s projected travel path, including at intersections and crosswalks


CEAV-VEH-033-v01

o Detecting and responding to vulnerable road users, such as pedestrians, cyclists, and scooters, approaching the vehicle’s projected travel path, including at intersections and crosswalks


CEAV-VEH-034-v01

o Providing a safe distance from vehicles, pedestrians, bicyclists, and scooters on the side of the road


CEAV-VEH-035-v01

o Providing a safe distance from vehicles, pedestrians, bicyclists, and scooters sharing the lane


CEAV-VEH-036-v01

o Decreasing speed when there is uncertainty regarding which action to take


CEAV-VEH-037-v01

o Detecting and responding to detours and other temporary changes in traffic patterns, such as people (including construction workers and police officers) directing traffic in unplanned or planned events. An acceptable response includes informing the human operator of the need to take manual control.


CEAV-VEH-038-v01

o Operating in normal rain, fog, and light snow conditions not deemed a weather emergency


CEAV-VEH-039-v01

o Deploying a wheelchair ramp RFP (pg. 28), contract (A.12)




040-v01 • The vehicles shall be programmed with the ability to reasonably

comply with local, state, and federal driving laws, regulations, ordinances, licenses, and certifications.


CEAV-VEH-041-v01

• The human operator shall have the ability to take manual control of the vehicle if deemed necessary.


CEAV-VEH-042-v01

• The vehicles shall be able to operate on the public roads as defined above in mixed traffic (integrated with other vehicles, trucks, bicyclists, pedestrians, etc.) without operator intervention, except in cases of extreme, abnormal conditions. Abnormal may include complete road closure due to a collision, unsafe road conditions (such as ice), or human-directed detours.


CEAV-VEH-043-v01

• The vehicles provided in the proposal must be accessible to those with disabilities. Onboard operators will be on board each vehicle during operations, and they may provide assistance to passengers beyond what the vehicle is independently capable of (such as securing a wheelchair or providing audible alerts). The Offeror shall identify its ability to comply with all applicable requirements of the Americans with Disabilities Act of 1990 (ADA), 42 U.S.C. 12101 et seq. and 49 U.S.C. 322; Section 504 of the Rehabilitation Act of 1973, as amended, 29 U.S.C. 794; Section 16 of the Federal Transit Act, as amended, 49 U.S.C. app. 1612; and the following regulations and any amendments thereto: o USDOT regulations, "Transportation Services for Individuals

with Disabilities (ADA)," 49 CFR. Part 37; o USDOT regulations, "Nondiscrimination on the Basis of

Handicap in Programs and Activities Receiving or Benefiting from Federal Financial Assistance," 49 CFR. Part 27;

o US. DOT regulations, "Americans With Disabilities (ADA) Accessibility Specifications for Transportation Vehicles," 49 CFR. Part 38;

o Department of Justice (DOJ) regulations, "Nondiscrimination on the Basis of Disability in State and Local Government Services," 28 CFR. Part 35;

o DOJ regulations, "Nondiscrimination on the Basis of Disability by Public Accommodations and in Commercial Facilities," 28 CFR. Part 36;

o General Services Administration regulations, "Construction and Alteration of Public Buildings," "Accommodations for the Physically Handicapped," 41 CFR. Part 101-19;

o Equal Employment Opportunity Commission (EEOC) "Regulations to Implement the Equal Employment Provisions of the Americans with Disabilities Act," 29 CFR. Part 1630;

o Federal Communications Commission regulations, "Telecommunications Relay Services and Related Customer Premises Equipment for the Hearing and Speech Disabled," 47 CFR. Part 64, Subpart F; and

o FTA regulations, "Transportation for Elderly and Handicapped Persons," 49 CFR Part 609.

RFP (pg. 23-24),

contract (A.2)

CEAV-DES-001-v01

• While the vehicles shall have a minimum capacity of 4 passengers (excluding the operator), higher (10+ person) capacity vehicles are preferred.




ID Requirement Source CEAV-DES-

002-v01 • The vehicles should be electric. RFP (pg. 24),

contract (A.2) CEAV-DES-

003-v01 • The service may provide Wi-Fi to onboard passengers. RFP (pg. 24),

contract (A.2) CEAV-DES-

004-v01 • The vehicles may have a screen on board, or a similar method of

communicating information to passengers. This may include information on their route and current location, as well as visuals on the data and images the sensors are receiving and making decisions based on. It may also provide real-time, relevant COTA vehicle information for passengers making transit connections (COTA already sends this data to the Operating System). This could also be the method for survey dissemination if the Offeror chooses.


CEAV-DAT-001-v01

• The Offeror shall agree to collect data and send this data to the Operating System via an API to inform the Data Sharing Requirements. Data will be stored for analytical purposes in the Operating System. Data that would be useful to potential passengers (such as real-time vehicle location information) will be shared via the Operating System at Smart Mobility Hub locations and in the Multi-Modal Trip Planning Application. o Vehicle route and schedule (GTFS) before launch and after

any changes or updates are made

RFP (pg. 25-26), contract (A.8)

CEAV-DAT-002-v01

o Real-time vehicle location information (AVL, GTFS Realtime) in real-time or near real-time


CEAV-DAT-003-v01

o Trip updates and service alerts in real-time or near real-time RFP (pg. 26), contract (A.8)

CEAV-DAT-004-v01

o Ridership (stop-level boardings and alightings) daily RFP (pg. 26), contract (A.8)

CEAV-DAT-005-v01

o Actual stop arrival and departure times daily RFP (pg. 26), contract (A.8)

CEAV-DAT-006-v01

o Vehicles miles traveled daily RFP (pg. 26), contract (A.8)

CEAV-DAT-007-v01

o Vehicle hours traveled (hours the vehicle is in service) daily RFP (pg. 26), contract (A.8)

CEAV-DAT-008-v01

o Number of route-trips served daily RFP (pg. 26), contract (A.8)

CEAV-DAT-009-v01

o Duration of each trip daily RFP (pg. 26), contract (A.8)

CEAV-DAT-010-v01

o Battery usage (such that it can be associated with weather, temperature, vehicle load, etc.) weekly


CEAV-DAT-011-v01

o Average vehicle speeds along each segment of the route weekly


CEAV-DAT-012-v01

o Count and duration of wheelchair ramp deployments weekly RFP (pg. 26), contract (A.8)

CEAV-DAT-013-v01

o Sensor and other telemetry data weekly RFP (pg. 26), contract (A.8)

CEAV-DAT-014-v01

o Navigation variances weekly RFP (pg. 26), contract (A.8)



ID Requirement Source CEAV-DAT-

015-v01 o Probe data (nRTK-enabled or similar) weekly RFP (pg. 26),

contract (A.8) CEAV-DAT-

016-v01 o Mechanical data (vehicle condition) weekly RFP (pg. 26),

contract (A.8) CEAV-DAT-

017-v01 o Disengagements/interventions by the operator, relative to the

amount of time spent in automated mode weekly RFP (pg. 26), contract (A.8)

CEAV-DAT-018-v01

o Any other logged events (hard stops, evasive maneuvers, unruly passenger behavior, etc.) weekly


CEAV-DAT-019-v01

o Conditions driven in (weather, congestion, etc.) weekly RFP (pg. 26), contract (A.8)

CEAV-DAT-020-v01

o Incident reports (Incidents include any collisions, and passenger behavior or other situations when an external entity is called upon for assistance) within one week following an incident


CEAV-DAT-021-v01

o Record of operational data exchanged (includes SPaT and MAP messages the vehicle receives, BSM it sends, etc.) weekly




Appendix E. EZ10 Acceptance Test Manual

Documents

Connected Electric Autonomous Vehicle (CEAV) Test Plan · 2019-10-08 · The purpose of this Test Plan is to document the strategic plan for testing and the criteria for acceptance