Driver License Automated Testing System

1 Department of Motor Vehicles | Vermont Agency of Transportation

Office of Purchasing & Contracting 10 Baldwin St [phone] 802-828-2211 Montpelier VT05633-7501 [fax] 802-828-2222 http://bgs.vermont.gov/purchasing

SEALED BID INFORMATION TECHNOLOGY REQUEST FOR PROPOSAL

FOR

Driver License Automated Testing System

Expected RFP Schedule Summary:

DATE POSTED December 21, 2012

QUESTIONS DUE January 16, 2013 4:30 PM EST

BIDDERS CONFERENCE January 25, 2013 1:00 PM EST

PROPOSALS DUE February 15, 2013 1:00 PM EST

FINALIST DEMONSTRATIONS Within 8 weeks of receiving proposals

SELECTION NOTIFICATION Within 3 weeks of finalist demonstrations

INDEPENDENT REVIEW To Be Determined

PROJECT STARTS Upon Fully Signed Contract

LOCATION OF BID OPENING: 10 Baldwin Street, Montpelier PLEASE BE ADVISED THAT ALL NOTIFICATIONS, RELEASES, AND AMENDMENTS ASSOCIATED WITH THIS RFP WILL BE POSTED AT: http://bgs.vermont.gov/purchasing/bids THE STATE WILL MAKE NO ATTEMPT TO CONTACT VENDORS WITH UPDATED INFORMATION. IT IS THE

RESPONSIBILITY OF EACH VENDOR TO PERIODICALLY CHECK http://bgs.vermont.gov/purchasing/bids FOR ANY

AND ALL NOTIFICATIONS, RELEASES AND AMENDMENTS ASSOCIATED WITH THE RFP.

PURCHASING AGENT: John McIntyre TELEPHONE: (802) 828-2210 E-MAIL: [email protected] FAX: (802) 828-2222

http://bgs.vermont.gov/purchasing

http://bgs.vermont.gov/purchasing/bids


mailto:[email protected]


Table of Contents

1 Overview and General Information ................................................................................................................................................. 4

1.1 Glossary of Terms ............................................................................................................................................................................ 4

1.2 Overview ......................................................................................................................................................................................... 4

1.3 Background ..................................................................................................................................................................................... 8

1.4 Single Point of Contact .................................................................................................................................................................... 8

1.5 Questions and Answers ................................................................................................................................................................... 8

1.6 Instructions for Bidders. .................................................................................................................................................................. 8

1.7 Schedule of Events ........................................................................................................................................................................... 8 1.7.1 Bidder's Conference (optional)……………..…………………………………………………………………………………………………………………………………………..9

2 Requirements 10

2.1 Costs of Preparation ...................................................................................................................................................................... 10

2.2 Quality ........................................................................................................................................................................................... 10

2.3 Worker’s Compensation; State Contracts Compliance Requirement ............................................................................................ 10

2.4 Invoicing ........................................................................................................................................................................................ 10

2.5 Cancellation ................................................................................................................................................................................... 11

2.6 Method of Award .......................................................................................................................................................................... 11

2.6.1 Evaluation Criteria.................................................................................................................................................................... 11

2.6.2 Evaluation Factors .................................................................................................................................................................... 11

2.7 Demonstration: ............................................................................................................................................................................. 12

2.8 Independent Review: ..................................................................................................................................................................... 12

2.9 Confidentiality ............................................................................................................................................................................... 13

2.10 Contract Terms .............................................................................................................................................................................. 13

2.11 Location of Work ........................................................................................................................................................................... 14

2.12 Statement of Rights ....................................................................................................................................................................... 15

2.13 Taxes ............................................................................................................................................................................................. 15

2.14 Order of Precedence ...................................................................................................................................................................... 15

2.15 Specification Change ..................................................................................................................................................................... 15

2.16 Amendments ................................................................................................................................................................................. 15

2.17 Non Collusion ................................................................................................................................................................................ 16

2.18 Business Registration .................................................................................................................................................................... 16

2.19 Contract Negotiation ..................................................................................................................................................................... 16

2.20 Software License and Maintenance Agreement Requirements ..................................................................................................... 16

3 Specific Bid Requirements ............................................................................................................................................................. 17

3.1 TECHNICAL PROPOSAL REQUIREMENTS ........................................................................................................................................ 17

3.2 Documentation .............................................................................................................................................................................. 20

3.3 Bidder Support / Training .............................................................................................................................................................. 20


3.4 Package Hardware and System Software ..................................................................................................................................... 20

3.5 Hardware/Software ...................................................................................................................................................................... 20

3.6 Custom Software ........................................................................................................................................................................... 20

3.7 Data ............................................................................................................................................................................................... 21

3.8 Software Licensing ........................................................................................................................................................................ 21

3.9 Application and Database Architecture ......................................................................................................................................... 21

3.10 Industry Standards and Certification ............................................................................................................................................. 21

3.11 Required Project Policies, Guidelines and Methodologies ............................................................................................................. 21

3.12 Measuremen: ................................................................................................................................................................................ 22

3.13 Functional Requirements ............................................................................................................................................................... 22

3.14 Value Added Features ................................................................................................................................................................... 53

3.15 Professional Services Requirements: ............................................................................................................................................. 53

3.16 Project Management..................................................................................................................................................................... 54

3.17 Staffing .......................................................................................................................................................................................... 54

3.18 Data Security ................................................................................................................................................................................. 54

3.19 Assumptions .................................................................................................................................................................................. 54

3.20 Performance Measures ................................................................................................................................................................. 54

4 Cost Proposal ................................................................................................................................................................................ 55

4.1 Non-recurring ................................................................................................................................................................................ 55

4.2 Recurring. ...................................................................................................................................................................................... 55

4.3 Price Guarantee. ............................................................................................................................................................................ 55

4.4 Maintenance Agreement............................................................................................................................................................... 55

4.5 New Releases. ............................................................................................................................................................................... 55

5 Bidder Response Content and Format .......................................................................................................................................... 56

5.1 Contents ........................................................................................................................................................................................ 56

5.2 Technical Proposal......................................................................................................................................................................... 56

5.3 Cost Proposal ................................................................................................................................................................................. 56

5.4 Financial Information. ................................................................................................................................................................... 56

5.5 Comprehensive Response. ............................................................................................................................................................. 56

5.6 Acknowledgment of Terms ............................................................................................................................................................ 56

5.7 Experience and Qualifications: ...................................................................................................................................................... 56

5.8 Financial Requirements: ................................................................................................................................................................ 57

5.9 Certificate of Compliance. ............................................................................................................................................................. 58

5.10 Workers’ Compensation; State Contracts Compliance Requirement; Self Reporting .................................................................... 58

5.11 Workers’ Compensation; State Contracts Compliance Requirement; Subcontractor Reporting. .................................................. 58

5.12 Offshore Outsourcing Questionnaire ............................................................................................................................................ 58

6 Submission Instructions and Checklist: ......................................................................................................................................... 58


6.1 Closing Date .................................................................................................................................................................................. 58

6.2 Submission Checklist ..................................................................................................................................................................... 59

6.3 Sealed Bid Instructions .................................................................................................................................................................. 59

6.4 Delivery Methods: ......................................................................................................................................................................... 59

7 Attachments: ................................................................................................................................................................................. 61

1 Overview and General Information

1.1 Glossary of Terms

ADA Americans with Disabilities Act

AOT Vermont Agency of Transportation

ATS Automated Testing System

CDL Commercial Driver’s License

COTS Commercial – Off-the-shelf

DMV Vermont Department of Motor Vehicles

IT Information Technology

LAN Local Area Network

PID Customer’s DMV Personal Identification Number

UL Underwriter’s Laboratories

VSA Vermont Statutes Annotated

WAN Wide Area Network

1.2 Overview

The State of Vermont (State) Office of Purchasing & Contracting on behalf of the Vermont Agency of

Transportation, Department of Motor Vehicles (DMV) is soliciting competitive sealed, fixed price proposals

(Proposals) for replacing its Automated Driver License Testing System (ATS) from qualified bidders. If a suitable

offer is made in response to this Request for Proposals (RFP), the DMV may enter into a contract (the Contract) to

have the selected Contractor perform all or part of the Work. This RFP details what is required to submit a


Proposal in response to this RFP, how the DMV will evaluate the Proposals, and what will be required of the

Contractor in performing the Work.

Structure and Scope of RFP

The DMV is seeking an automated driver license testing system to replace the existing one now in use. The

new system must operate at each DMV office throughout the State and be flexible, allowing DMV

management to author new questions and answers and update visuals as needed. The new system must

be able to randomize questions for each test, to record the results, to keep records and allow for standard

and ad hoc report generation from a central database. The system must be capable of presenting the

questions in several different languages; test results must be made available to the tester immediately.

Personally identifiable information must be treated securely. The proposed system must also be able to be

interfaced with current or future applications to allow the DMV to integrate this system with others (such

as a scheduling system). The selected Contractor will provide hardware specifications for the system. The

DMV prefers to purchase the hardware through existing State contracts although proposals will be

considered for purchasing hardware from the selected Contractor.

The Automated Testing System (ATS) Replacement Project shall consist of replacing the current ATS in all DMV

facilities with new software, hardware and testing station containing the most recent proven and up-to-date

technology available.

The System will be installed in 2 phases:

Phase 1 installation will be to establish a test system and will consist of software, test unit, hardware in the DMV

Main Exam Office. All required hardware and software to perform development, testing and acceptance is

included. This will include application customization and data conversion of files from the existing system to the

new system.

Phase 2 will consist of installation at all remaining branch offices for pilot roll out purposes.

Bidder will develop and submit a detailed project plan that includes, but is not limited to, the following:

Detailed plans and methodology for each phase of customization, installation, testing, and acceptance.

Detailed training plan and strategy for each phase and all identified AOT staff.

Detailed payment schedule upon Agency acceptance of software, hardware, installations, and training at the

completion of each phase.

Detailed schedule for Phase 1 installation in the DMV Central Office of equipment, software design, construction,

testing, and acceptance.

Detailed schedule for Phase 2 installations for software and testing equipment in all branches, test and

acceptance periods.


Coordination of new equipment installation in field offices with no or minimal office down time.

Detailed schedule coordination with the DMV for all installations.

Objective of RFP:

The DMV desires a solution that is technologically current and tested. It should include software, a database, and

proposed hardware. The DMV desires to take advantage of the current infrastructure that we have in place. A

description of this infrastructure is provided in Appendix F.

The selected contractor will work with DMV staff to tailor the system to our specific needs if necessary, and

convert existing test questions, video, audio files and history records from the existing system to the new solution.

The selected contractor will provide installation services, training, and then on-going support after installation.

Department Background and Philosophy:

The DMV serves the general public from the Central Office, five fixed branch offices, and two mobile vans and

strives to provide the highest level of customer service. Members of the public schedule appointments in advance

for tests. Several people may be testing at the same time, at the same location. This depends on the office size

and location. Requirements for each testing site will be described below. Approximately 30,000 exams are given

a year.

The five branch offices are located in Bennington, South Burlington, Newport, Rutland and Springfield.

Additionally there are two mobile vans in operation that travel to five other locations performing functions similar

to those administered at the branches. More information on branch and mobile van locations can be found at

http://dmv.vermont.gov/locations .

Quick, easy service is one of the main goals of the Department.

Compatibility with Standards and Existing Technology:

The proposed system must conform to the Federal ADA standards

Current Business Environment:

Funds for this project are available and the DMV will proceed with a system replacement, as long as a suitable ATS

is proposed.

The DMV is organizationally part of the Vermont AOT (AOT). Technical support (application development,

networking and PC support) are provided by the AOT’s central Information Technology (IT) section. The IT section

will work with the selected Contractor to provide servers (virtual where possible), database support, security,

system installation, system updates and helpdesk support.

The infrastructure that the system will run on includes networking, redundant servers and a MS SQL Server

database cluster. The current testing stations (furniture) will need to be replaced. The PC Support staff will

provide the base system set-up based on the bidder specifications. They will also assist in installation of the

application on the workstations. Contractor will be responsible to install any application components on the

http://www.aot.state.vt.us/conadmin/Documents/ProceduresForSelectingContractorsandSpecificationsforContractorServices.pdf


server, with oversight from the IT Application staff, and database installation will be done in conjunction with the

IT Database Administration staff. Implementation of the new system must be done with minimal impact to on-

going operations.

Installation of workstation furniture will be done by the Contractor with oversight of DMV staff.

CURRENT SYSTEM

The current DMV ATS consists of 32 Test Units, 8 Exam Consoles and 1 Administrator Console spread out among 1

central office, 5 branch offices and 2 mobile units. Each field office Exam Console maintains a software installation

containing the test application, questions and answers. With each completed test, a score is displayed, provided

to the applicant and written on the license application by the examiner.

Note: new configuration to consist of: 34 test units, (1 central office with five workstations, 5 branch offices; one

with five workstations, four offices with four per office and two mobile offices with four per mobile for a total of

34) and one support station.

The following chart represents the current number of questions that are used on a particular test type, the

current pool size of questions to draw from, and other information pertinent to particular tests.

Test Question Specifications

Automated Testing System

Test Questions Pool Domains Max Wrong / Min Right

(80% to pass)

Learners 20 53 2 4 / 16

Motorcycle 25 97 12 5 / 20

CDL

General 50 265 21 10 / 40

Air Brake 25 67 7 5 / 20

Combination 20 72 3 4 / 16

HazMat 30 87 5 6 / 24

Passenger 20 51 6 4 / 16

Doubles /

Tripples

20 54 5 4 / 16

Tanker 20 51 7 4 / 16

School Bus

Type I {Federal}

(CDL) 31, Nine Fill In The Blank Questions

Twenty Multiple Choice Questions

Two Essay Questions

Type II {State} (non-CDL) 37, Twenty-Six True False Questions

Nine Fill In The Blank Questions

Two Essay Questions

DMV currently processes an average of 30,000 examinations per year. Of these tests, 16.6% or 4,971

test/examinations are CDL tests and 25,029 or 83.4% are non-CDL tests.


1.3 Background

The State of Vermont would like to augment internal IT resources with high quality, contractor -provided turnkey

system and personal services. We are interested in identifying contractors that can provide an ATS and services to

implement and train State employees on use and support. Additional support may also be needed from the

contractor in the form of an annual support contract.

1.4 Single Point of Contact

All communications concerning this Request for Proposal (RFP) are to be addressed in writing to the attention of:

John McIntyre, Purchasing Agent, State of Vermont, Office of Purchasing & Contracting, 10 Baldwin St -

Montpelier, VT 05633-7501. John McIntyre, Purchasing Agent is the sole contact for this proposal. Actual contact

with any other party or attempts by bidders to contact any other party could result in the rejection of their

proposal.

E-mail Address: [email protected]

1.5 Questions and Answers

Any bidder requiring clarification of any section of this proposal or wishing to comment or take exception

to any requirements or other portion of the RFP must submit specific questions in writing no later than

January 16, 2013 @ 4:30 P.M. Questions may be e-mailed to [email protected]. Any objection to

the RFP or to any provision of the RFP, that is not raised in writing on or before the last day of the question

period is waived. At the close of the question period a copy of all questions or comments and the State's

responses will be posted on the State’s web site http://bgs.vermont.gov/purchasing/bids. Every effort will

be made to have these available as soon after the question period ends, contingent on the number and

complexity of the questions.

1.6 Instructions for Bidders: see sections 5 and 6.

1.7 Schedule of Events

The expected timetable, including the Proposal due date and other important dates, are set forth below.

The name of any Bidder submitting a proposal shall be a matter of public record on the Proposal Due Date.

RFP Published December 21, 2012

Questions Due January 16, 2013 4:30 p.m. ET

Answers to Questions Posted January 23, 2013 4:30 p.m. ET

Optional Bidders Conference Must pre-register to attend See Section 1.7.1

January 25, 2013 1:00 P.M. EST,

Proposal Due (see section below for detailed

instructions on Proposal Format and submission

Feb 15, 2013 1:00 p.m. ET





instructions)

Finalist Selection / Notification Within 5 weeks of receiving proposals

Finalist Demonstrations Within 3 weeks of finalist notification

Recommendation for Award Within 3 weeks of finalist demonstrations

Independent Review Completed

Following the selection of a proposal for contract

award, the selected proposal will be the subject of an

independent review before a contract can be

completed if applicable. The time required for this

process is, at a minimum, 3 weeks.

To Be Determined.

1.7.1 Bidders Conference (optional)

You must call or email your intent to attend in person, or call into, this Conference session no later than 2pm EST on January 22, 2013. Call John McIntyre at 802-828-2210 or email [email protected]. The Conference will be in Montpelier Vermont, and available by conference call to those unable to attend. Location details and call-in information will be provided to those who register.

The bidder(s) shall be solely responsible for all expenses incurred in the attendance of the Bidder’s conference.



2 Requirements

2.1 Costs of Preparation

The bidder(s) shall be solely responsible for all expenses incurred in the preparation of a response to this

RFP and shall be responsible for all expenses associated with any presentations or demonstrations

associated with this request and/or any proposals made.

2.2 Quality

If applicable, all products provided under these agreements will be new and unused, unless otherwise

stated. Factory seconds or remanufactured products will not be accepted unless specifically requested by

the purchasing agency. All products provided by the Contractor must meet all federal, state, and local

standards for quality and safety requirements. Products not meeting these standards will be deemed

unacceptable and returned to the Contractor for credit at no charge to the State.

2.3 Worker’s Compensation; State Contracts Compliance Requirement

The Department of Buildings and General Services in accordance with Act 54, Section 32 of the Acts of

2009 and for total projects costs exceeding $250,000.00, requires bidders comply with the following

provisions and requirements.

(a) (1) Bidder is required to self report detailed information including information relating to past violations, convictions, suspensions, and any other information related to past performance and likely compliance with proper coding and classification of employees requested by the applicable agency.

The bidder is required to report information on any violations that occurred in the previous 12 months.

(a) (2) Bidder is required to provide a list of subcontractors on the job along with lists of subcontractor’s subcontractors and by whom those subcontractors are insured for workers’ compensation purposes. Include additional pages if necessary. This is not a requirement for subcontractor’s providing supplies only and no labor to the overall contract or project.

In order for a bidder’s response to be considered valid, bidders must complete and submit the following

two (2) forms at time of bid:

Self Reporting

Subcontractor Reporting

2.4 Invoicing

All invoices are to be rendered by the contractor on the contractor’s standard billhead and forwarded to

the DMV Project Manager. Details such as name, title & address will be determined during negotiations.


The bidder’s proposal must clearly specify the address for submitting payments. All payments are to be

based on State of Vermont’s acceptance of agreed to, fixed price deliverables.

2.5 Cancellation

The State specifically reserves the right to cancel the contract, or any portion thereof, if, in the opinion of

its Commissioner of Buildings and General Services, the services or materials supplied by the contractor are

not satisfactory or are not consistent with the terms of the contract

2.6 Method of Award

Awards will be made in “the best interest of the State of Vermont”. The DMV may award one or more

contracts and reserves the right to make additional awards to other compliant bidders at any time during

the first year of the contract if such award is deemed to be in the best interest of the State.

2.6.1 Evaluation Criteria

In general, bids are awarded based on “the best interest of the State of Vermont”.

2.6.2 Evaluation Factors

EVALUATION FACTORS Factor Weight

(Maximum Points)

1. Experience 20

Documented and demonstrated prior experience with similar testing applications, especially in other states.

DMV-related implementation references.

Knowledge of subject matter.

Resumes of staff assigned

Stability of company

2. Adequate Resources/Staffing to Accomplish Work 15

3. Project Plan and Schedule 10

4. Implementation plan 15

5. Proposed Solution 30


Technically current

Ease of use

Meets mandatory requirements Pass/Fail

Accumulation of Desirable features

On-going Support

7. Response Pass/fail

Completeness

Clarity

8. References 10

9. Demonstration (if requested) 20

TOTAL

Once the technical proposal is discussed and ranked, the cost proposal will be reviewed for consistency with, and in light of, the evaluation of the technical proposal. DMV reserves the right to seek clarification of any proposal submitted and to select the proposal considered to best promote the public interest.

2.7 Demonstration:

After initial evaluation of the proposals, the DMV may request demonstrations by the highest scoring bidders. This will help the DMV measure the rated factors. The DMV will factor information presented during demonstrations into the evaluation scoring. Bidders will be responsible for all costs associated with providing the demonstration. Demonstrations will take place in Montpelier at the DMV Central Office.

2.8 Independent Review:

Per Vermont statute, all information technology contracts with a value of $500,000 or more must undergo an “independent review” by an uninvolved party who is engaged by the State Chief Information Officer to review the decision of the contracting agency in its selection of a particular information technology proposal. The cost of the independent review is borne by the State Sponsor Agency. This review includes the following elements:

An acquisition cost assessment

A technology architecture review

An implementation plan assessment

A cost/benefit analysis


2.9 Confidentiality

The successful response will become part of the contract file and will become a matter of public record, as

will all other responses received. If the response includes material that is considered by the bidder to be

proprietary and confidential under 1 VSA, Chapter 5, the bidder shall clearly designate the material as such,

explaining why such material should be considered confidential. The bidder must identify each page or

section of the response that it believes is proprietary and confidential with sufficient grounds to justify

each exemption from release, including the prospective harm to the competitive position of the bidder if

the identified material were to be released. Under no circumstances can the entire response or price

information be marked confidential. Responses so marked may not be considered.

2.10 Contract Terms

The selected bidders will sign a contract with the DMV to provide the items named in their responses, at

the prices listed. This contract will be subject to review throughout its term. The DMV will consider

cancellation upon discovery that a Contractor is in violation of any portion of the agreement, including an

inability by the contractor to provide the products, support, and/or service offered in their response. The

State reserves the right to purchase hardware or software recommended in the contractor’s proposal from

any State contract in force.

Bidders planning to submit a bid are advised of the following:

1. The selected contractor will sign a contract with the State to provide the items named in its response, at the prices listed. Minimum support levels, terms, and conditions from this RFP, and the contractor’s response, will be in the contract. PLEASE NOTE THAT THE STATE WILL NOT ACCEPT THE CONTRACTOR’S STANDARD FORM IN LIEU OF THE STANDARD STATE CONTRACT PROVISIONS.

2. The State of Vermont expects the bidder and their legal counsel to carefully review and be prepared to be bound by the Standard State Provisions for Contracts and Grants , the Procedures for Selecting Contractors and Specifications for Consultant Services, and the General Special Provisions all of which comprise Attachment C .

3. If the bidder wishes to propose an exception to any term or condition in the Standard State Provisions for Contracts and Grants, the Procedures for Selecting Contractors and Specifications for Consultant Services or the General Special Provisions it must notify the DMV in its response to the RFP. Failure to note exceptions will be deemed to be acceptance of the Documents as attached to the RFP. If exceptions are not noted in the Proposal but raised during contract negotiations, the DMV reserves the right to cancel the negotiation if deemed to be in the best interests of the State of Vermont.

4. All contracts resulting from this RFP are subject to review and approval by the Attorney General, Vermont’s Chief Information Officer, the Secretary of Transportation and the Secretary of Administration. A separate software license and maintenance support agreement will be required for all major software, if any, proposed for this project.

5. Once awarded, the term of the Contract will be from the date the contract is fully signed forward for a term of two years. The DMV may renew this Contract for two additional one year periods subject to and contingent upon the discretionary decision of the Vermont Legislature to appropriate funds for this Contract in each new fiscal year. The DMV may renew all or part of this Contract subject to the satisfactory performance of the Contractor and the needs of DMV. The bidder should guarantee their rate offerings, over the term of the contract, are comparable to other customers of similar size and


requirements. If offerings are rendered to a comparable customer which improve the pricing agreed to in the contract, the bidder agrees to apply those same discounts and offerings to the State of Vermont.

6. The State of Vermont has no legal authority to indemnify a bidder and will not consider additional limitations of liability either for the primary contract or any related software agreement that the Bidder wishes the State to sign, and those issues are not negotiable. Bidders who are not able to legally enter into a contract under those conditions should not submit a bid.

7. The State may require selected bidders to furnish, without additional cost, a performance, payment or bid bond or negotiable irrevocable letter of credit or other form of security for the faithful performance of the Contract. Such bond or other security shall be in the form prescribed by the State. The State may issue this requirement to finalists, at the time the contract is signed, or at any time during the term of the Contract.

2.11 Location of Work

Application customization, data conversion and initial test system setup and testing will be done in the Montpelier

main DMV office. To minimize office disruption, installation at each fixed office location may need to be done

after office hours or on a weekend. Installation work will be required at the locations listed below:

DMV Main Office (Montpelier)

DMV Branch Office (Bennington)

DMV Branch Office (South Burlington)

DMV Branch Office (Newport)

DMV Branch Office (Rutland)

DMV Branch Office (Springfield)

Mobile Van sites are listed below. There are two mobile units, each one contains the same equipment as the fixed location offices but since the equipment will be frequently moved, these systems should be proposed accordingly. The equipment in the mobile units is carried into a building where staff set up to serve customers for the day. Network connections are in each of these locations. Testing will need to be done from each van location, listed below.

DMV Mobile Location (Dummerston)

DMV Mobile Location (Middlebury)

DMV Mobile Location (St Albans)

DMV Mobile Location (St Johnsbury)

DMV Mobile Location (White River Junction)

As a general rule, project work will be done in Montpelier, VT. The Contractor will be required to work on-site in

(such site or sites in Montpelier, VT. as may be identified by the DMV) where space will be provided, however

travel to other State facilities may be needed and the Contractor will be responsible for such travel using their


own mode of transportation. Occasional exceptions to this rule may be established by mutual agreement

between the Contractor and the DMV Project Manager.

Where applicable, the DMV will provide a project facility with desks, telephone, LAN connections, and printers. The DMV will provide desktop PCs and/or laptops for use during the project. If specific laptop computers or other mobile peripheral devices are required by the Contractor then the Contractor must provide their own equipment and will be given the appropriate support by the DMV. Contractor will be provided support by the DMV in setting up any accounts or connections required (i.e. State email system, network connectivity, network printing etc.) and Contractor will have access to State phones for use in project related business calls. The State will not pay Contractor ‘s cell phone bills.

The successful bidder will be required to complete a non-disclosure agreement and a Driver Privacy Protection Act agreement (DPPA), Attachment G.

2.12 Statement of Rights

The DMV reserves the right to obtain clarification or additional information necessary to properly evaluate

a proposal. Bidders may be asked to give a verbal presentation of their proposal after submission. Failure

of bidder to respond to a request for additional information or clarification could result in rejection of that

bidder's proposal. To secure a project that is deemed to be in the best interest of the State, the State

reserves the right to accept or reject any and all bids, in whole or in part, with or without cause, and to

waive technicalities in submissions. The State also reserves the right to make purchases outside of the

awarded contracts where it is deemed in the best interest of the State.

2.13 Taxes

Most State purchases are not subject to federal or state sales or excise taxes and must be invoiced tax free.

An exemption certificate will be furnished upon request covering taxable items. The Contractor agrees to

pay all Vermont taxes which may be due as a result of this order. If taxes are to be applied to the purchase

it will be so noted in the response.

2.14 Order of Precedence

The order of precedence for documentation will be the State of Vermont Standard Contract Form and

attachments, the bid document and any amendments, and the bidder’s response and any amendments.

This is not negotiable.

2.15 Specification Change

Any changes or variations in the specifications must be received in writing from the Office of Purchasing &

Contracting. Verbal instructions or written instructions from any other source are not to be considered.

2.16 Amendments

No changes, modifications, or amendments in the terms and conditions of this Contract shall be effective

unless reduced to writing, numbered, and signed by the duly authorized representative of the State and

Contractor.


2.17 Non Collusion

The State of Vermont is conscious of and concerned about collusion. It should therefore be understood by

all that in signing bid and contract documents they agree that the prices quoted have been arrived at

without collusion and that no prior information concerning these prices has been received from or given to

a competitive company. If there is sufficient evidence to warrant investigation of the bid/contract process

by the Office of the Attorney General, all bidders should understand that this paragraph might be used as a

basis for litigation.

2.18 Business Registration

To be awarded a contract by the State of Vermont a bidder must be (except an individual doing business in

his/her own name) registered with the Vermont Secretary of State’s office

http://www.sec.state.vt.us/tutor/dobiz/forms/fcregist.htm and must obtain a Contractor’s Business

Account Number issued by the Vermont Department of Taxes

http://www.state.vt.us/tax/pdf.word.excel/forms/business/s-1&instr.pdf

2.19 Contract Negotiation

Upon completion of the evaluation process, the DMV may select one or more bidder with which to

negotiate, if necessary, a contract, based on the evaluation findings and other criteria deemed relevant for

ensuring that the decision made is in the best interest of the State of Vermont. In the event the DMV is

successful in negotiating with the bidder , the State will issue a notice of award. In the event DMV is not

successful in negotiating a contract with a selected bidder, the DMV reserves the option of negotiating

with another bidder.

2.20 Software License and Maintenance Agreement Requirements

The bidder shall provide the DMV with a Contract or Agreement for any System licenses and maintenance

support that is part of the proposed requirements. Failure to provide the license and maintenance

agreements as part of the RFP response may result in rejection of the bidder’s proposal.

As already noted, the State of Vermont has no legal authority to indemnify a bidder, and will not consider

additional limitations of liability either for the primary contract or any related software agreement that the

Bidder wishes the State to sign, and those issues are not negotiable. Bidders who are not able to legally

enter into a contract under those conditions should not submit a bid.

http://www.sec.state.vt.us/tutor/dobiz/forms/fcregist.htm

http://www.state.vt.us/tax/pdf.word.excel/forms/business/s-1&instr.pdf


3 Specific Bid Requirements:

3.1 TECHNICAL PROPOSAL REQUIREMENTS

For each paragraph, bidder must respond/acknowledge to the requirements described.

3.1.1 Provide all application software, application customization, hardware specifications, labor, workstations, materials, training, training manuals, training supplies, technical support, warranties, and low voltage wiring between ATS components.

3.1.2 Develop and provide the customization/data migration plan, system testing plan, and project schedule in collaboration with DMV.

3.1.3 Provide the following work products including but not limited to:

Work Plan /Project Schedule

User manual / User Documentation

Test Plan / Test Results

Training Plan

Implementation Plan

Installation Plan and installation steps.

User guides

3.1.4 Acknowledgement of understanding of the cooperative nature bidder will enter into with the Agency’s I.T staff for system installation and on-going administration of this system.

3.1.5 Bidder shall release to DMV all proprietary licenses, releases, software, and equipment. DMV reserves the right to purchase, lease or contract license and software if it is in the best interest of DMV to do so.

3.1.6 Provide a complete System (software and proposed hardware) that is compatible and meets AOT DMV business and technical requirements.

3.1.7 Bidder shall convert all DMV test questions, answers and other related data including the entire current data bank containing historical information onto the new ATS.

3.1.8 Provide an ATS capable of meeting DMV acceptance testing in the agreed upon project phase and timeframe.

3.1.9 Supply copy of all Software License and/or Maintenance Agreements with proposal.


3.1.10 Provide labor, hardware, parts (e.g., cables, furniture, and fasteners etc) for proper installation and operation of the system. These costs must be included in the Cost Proposal.

3.1.11 Provide complete installations and all training at DMV Central Office and field offices throughout the state. Times and locations will be mutually agreed by Agency and Contractor.

3.1.12 All ATS and system related training for DMV staff shall be performed only by the awarded Contractor’s employees who have extensive knowledge and experience.

3.1.13 Provide continued training throughout the term of the contract at times and locations as mutually agreed.

3.1.14 Provide continued technical support throughout the term of Contract and mutually agreed upon maintenance periods, working with the Agency IT Helpdesk. Submit options for on-going support agreements.


Exhibit A

Current Equipment List / Required Equipment List by Location

Bidder is required to propose the number of workstations in the Total (Prop) column for each location.

REGION Test Units

Office Total

(current)*

Total

(prop)*

ADA Compliant Exam. Console Admin.

Console

Montpelier 5 5 1 1 1

Burlington 5 5 1 1 0

Rutland 4 4 1 1 0

Newport 4 4 1 1 0

Springfield 4 4 1 1 0

Bennington 3 4 1 1 0

Southern

Mobile Van

3 4 1 1 0

Northern

Mobile Van

3 4 1 1 0

Support

Station

0 1 1

TOTALS 31 35 8 8 2


3.2 Documentation:

Appropriate documentation/manuals and training material must be provided for on-going use and support of this system on CD or DVD. Documentation that might be changed by DMV over time must be provided in modifiable format, preferably Microsoft Office products (Word, Excel, etc).

3.3 Bidder Support / Training:

The DMV desires to be as self sufficient as practical in the administration, use and technical support of this system. To that end, training is needed for system administrators, system users and technical staff for installation, maintenance and trouble shooting. Trouble calls will be reported to the DMV helpdesk where it will be triaged. The DMV Helpdesk will check the hardware and network connections, and determine if any changes have been made to the user PC configuration (such as software updates) and will contact the Contractor when necessary. Contractor will be responsible to maintain and support all components they provide

3.4 Package Hardware and System Software:

The DMV prefers an off-the-shelf system (COTS) if one satisfies our requirements. Describe your entire system, and how each component interacts and supports the other components of the system, both hardware and software. Where possible, our existing infrastructure (Network, physical database servers, and virtual applications servers) should be used. Contractor will work with the DMV technical staff to integrate with the AOT’s Active Directory; develop a backup/restore and virus protection program that is consistent with the Agency’s current practices.

3.5 Hardware/Software

While the bidder may propose to provide hardware and software as part of their proposal the DMV

reserves the right to procure hardware and commercial off-the-shelf software from other sources when it

is in the best interest of the DMV to do so. The bidder must provide hardware specifications for this

system.

3.6 Custom Software

The DMV shall solely own any custom software, including, but not limited to application modules developed to integrate with a COTS product, source-codes, maintenance updates, documentation, training materials, and configuration files, developed under any Scope of Work (SOW). Custom software developed for the DMV shall not be reused, resold, re-licensed, or repurposed by a bidder without written permission from an authorized representative of the State’s CIO.

Upon the Contractor’s voluntary or involuntary filing of bankruptcy or any other insolvency proceeding, Contractor’s dissolution, Contractor’s merger with or acquisition by another company or contractor, discontinuance of support of any software or system, the Contractor shall convey to the DMV all rights, title, and interests in all custom software, software source codes, and all associated Software Source Code Documentation. For all custom software provided to the DMV pursuant to contract(s) awarded as a result of this RFP, the Contractor shall either provide the source code directly


to the DMV in a form acceptable to the DMV, or deliver two copies of each software source code and Software Source Code Documentation to a DMV-approved escrow agent.

3.7 Data

Data and derived data products (including aggregated, “de-identified”, or “randomized” data) collected, manipulated, or directly purchased as part of a SOW shall become the exclusive property of the DMV. The DMV is considered the custodian of the data and shall determine the use, access, distribution and other conditions based on appropriate State statutes and regulations.

Licensed and/or copyrighted data shall be governed by the terms and conditions identified in the terms of agreement or the license.

3.8 Software Licensing:

Include all software licensing options for any proprietary software proposed as part of this system; explain the advantages and/or disadvantages of each option, and the recommended option based on information in this document. Include the cost for each licensing option in the Cost Proposal, no cost information shall be provided in the technical proposal.

3.9 Application and Database Architecture:

Describe your application and database architecture. As mentioned earlier, the DMV has the infrastructure in place to create virtual servers (using VMware) and a database cluster using MS SQL Server 2008.

Describe how this application may be interfaced with other applications the DMV currently uses or may use in

the future.

3.10 Industry Standards and Certification:

The ATS must be compliant with American Association of Motor Vehicle Administrators (AAMVA) Guidelines for

Knowledge and Skills Testing and Federal Motor Carrier Safety Administration (FMCSA) Regulation 49 CFR and the

Motor Carrier Safety Improvement Act (MCSIA). (Attachment I)

3.11 Required Project Policies, Guidelines and Methodologies

The Contractor shall be required to comply with all applicable laws, regulations, policies, standards and guidelines

affecting information technology projects, which may be created or changed periodically. It is the responsibility of

the Contractor to insure adherence and to remain abreast of new or revised laws, regulations, policies, standards

and guidelines affecting project execution. Agency specific confidentiality and privacy policies, such as Health

Insurance Portability and Accountability Act (HIPAA) may apply. These may include, but are not limited to:

The State’s Information Technology Policies & Procedures at: http://dii.vermont.gov/Policy_Central

http://dii.vermont.gov/Policy_Central


The State’s Department of Information and Innovation (DII) Enterprise Project Management Office (EPMO) process for managing Information Technology projects at: http://dii.vermont.gov/pm/process

The State’s Record Management Best Practice at: http://vermont-archives.org/records/standards/pdf/RecordsManagementBestPractice.pdf

The State Information Security Best Practice Guideline at: http://vermont-archives.org/records/standards/pdf/InformationSecurityBestPractice_Eff.20090501.pdf

The State Digital Imaging Guidelines at http://vermont-archives.org/records/standards/pdf/ImagingGuideline2008.pdf

The State File Formats Best Practice at http://vermont-archives.org/records/standards/pdf/FileFormatsBestPractice_Eff.20071201.pdf

The State File Formats Guideline at http://vermont-archives.org/records/standards/pdf/FileFormatsGuideline2008.pdf

The State Metadata Guideline at http://vermont-archives.org/records/standards/pdf/MetadataGuideline2008.pdf

3.12 Measurement:

The system needs to be able to provide testing for the maximum number of test takers at each location simultaneously. Response time to the test taker must be 2 seconds from time selecting an answer to being ready for the next question. Test results must be made available to the local test administrator within one minute after the test has been completed. Test results must be centrally available to DMV management by end of closing each day.

3.13 Functional Requirements:

MANDATORY AND DESIRABLE REQUIREMENTS

The DMV has developed a Requirements Outline with two assigned rankings. The requirements are configured

into 15 requirement sub-categories. The rankings are “Mandatory” and “Desirable.” Bidders are required to

identify the capabilities of their system for each Mandatory and Desirable Requirement.

The requirements ranked as Mandatory are the absolute minimum requirement for the ATS. DMV will reject any

Proposal which fails to meet one or more Mandatory Requirements.

http://dii.vermont.gov/pm/process

http://vermont-archives.org/records/standards/pdf/RecordsManagementBestPractice.pdf

http://vermont-archives.org/records/standards/pdf/RecordsManagementBestPractice.pdf

http://vermont-archives.org/records/standards/pdf/InformationSecurityBestPractice_Eff.20090501.pdf


http://vermont-archives.org/records/standards/pdf/ImagingGuideline2008.pdf

http://vermont-archives.org/records/standards/pdf/ImagingGuideline2008.pdf



http://vermont-archives.org/records/standards/pdf/FileFormatsGuideline2008.pdf

http://vermont-archives.org/records/standards/pdf/FileFormatsGuideline2008.pdf

http://vermont-archives.org/records/standards/pdf/MetadataGuideline2008.pdf

http://vermont-archives.org/records/standards/pdf/MetadataGuideline2008.pdf


The requirements ranked as Desirable identify desired additional requirements. There are 42 listed Desirables.

Bidder’s capabilities for Desirables will be considered in the Evaluation Factors explained in section 2.6.2.

For each line item in the chart below, Bidder shall provide a detailed statement by “Requirement #” as specified in

Section 5 below describing how their system meets or exceeds the minimum requirement.

Require-

ment # Category Requirement Agency Notes /

Assumptions / Constraints

Rank

1. 1.Encompassing ATS must be compatible with

DMV Systems (VT/AOT servers

and software and legacy DMV

Systems)

Mandatory

2. 1.Encompassing All ATS System components

must be of current production

models

Mandatory

3. 1.Encompassing All proposed ATS technical

components must have been

manufactured within one year of

installation (excluding printers)

Mandatory

4. 1.Encompassing All proposed ATS components

must be new and previously

unused

Mandatory

5. 1.Encompassing The number of ATS test stations

and Examiner access points must

be expandable without ATS

software or hardware upgrades

Mandatory

6. 1.Encompassing The ATS must have the capacity

to operate at 6 testing stations per

Examiner access point

Mandatory

7. 1.Encompassing All kiosks/furniture must have

been manufactured within one

year of installation

Mandatory

8. 1.Encompassing All ATS components must be

from a product line that has been

available on the open market for a

minimum of one year prior to the

date of the contract

Mandatory


Require-



Rank

9. 1.Encompassing The ATS should have a

sustainable lifespan of at least ten

years, with an upgrade path to

prevent obsolescence for a period

of five years

Desirable

10.

2. Hardware –

General The ATS power cords must be

equipped with three-prong plug-

ins and meet Underwriters

Laboratory Standards

Mandatory

11. 2. Hardware –

General One testing station in each office –

eight in total– must conform to

American Disabilities Act (ADA)

requirements

Mandatory

12. 2. Hardware –

General Propose all hardware necessary for

replacing the ATS which, at a

minimum, must include:

2 access points for Administrators

(currently an individual

Administrator console)

8 access points for Examiners

(currently individual Examiner

consoles)

34 testing stations

1 Support Station

Mandatory

13. 2. Hardware –

General The ATS testing station must use

securely attached private listening

devices for Audio tests

Mandatory

14. 2. Hardware -

General The ATS must use a digitized

recording system for audio

recordings (i.e., no tape drives or

other mechanisms prone to wear)

Mandatory

15. 2. Hardware -

General The ATS must fit within the space

currently allocated in field offices

for testing stations and Examiner

Consoles

Maximum Size Dimensions for

non ADA Units – 48” High x 35”

Wide x 31” Deep

Mandatory

16. 2. Hardware -

General The ATS must be installed

without site modifications

Mandatory

17. 2. Hardware -

General The ATS must have all cabinets

constructed of non-conductive

material to minimize the chance of

electrical shock to customers or

employees

Mandatory


Require-



Rank

18. 2. Hardware -

General The ATS must have all surfaces

free from sharp edges and burrs

and be constructed of high quality,

durable materials and free of any

design defects.

Mandatory

19. 2. Hardware -

General The ATS must be quiet:

No audible beeps or voice

commands from the test station

(note: voice should be available

through headset)

Operating noise level (e.g., fans,

drives) that do not exceed 40

decibels

Mandatory

20. 2. Hardware -

General The ATS consoles/testing stations

must allow access to internal

components for repair and

maintenance without interfering

with the use of other testing

stations

Mandatory

21. 2. Hardware -

General The proposed ATS hardware must

include all communications

components and hardware

Mandatory

22. 2. Hardware -

General The proposed ATS must operate

on 110/120 VAC, 60 Hz electrical

power

Mandatory

23. 2. Hardware -

General All ATS wiring and connections

must be secured from customers

(i.e., customers are unable to

access or be shocked by the testing

stations)

Mandatory

24. 2. Hardware -

General The ATS must use removable

plug-in connectors for all

electrical connections between

sub-assemblies, control panels,

etc.

Mandatory

25. 2. Hardware -

General The ATS must prohibit exposed

voltages when access panels are

removed

Mandatory

26. 2. Hardware -

General The ATS must report testing

station hardware/software failures

to the Examiner immediately after

power-up

Mandatory


Require-



Rank

27. 2. Hardware -

General For each console/testing station,

the ATS must provide a prompt

for the Examiner that indicates

"ready" status (i.e., functioning

correctly, available for testing,

tests and software up-to-date, etc.)

Mandatory

28. 2. Hardware -


must be secured against

unauthorized access to internal

components (secured against

tampering with internal hardware)

Mandatory

29. 2. Hardware -

General The ATS must use commercial

'off-the-shelf' hardware (i.e., no

proprietary hardware)

Mandatory

30. 2. Hardware -

General All testing stations should be

designed to allow for installation

of visual privacy barriers

Mandatory

31. 2. Hardware -

General All testing stations should be

designed to allow for installation

of visual privacy barriers

Mandatory

32. 2. Hardware -

General The ATS should incorporate

modular construction for ease of

maintenance wherever possible

Desirable

33. 2. Hardware -

General The ATS should have 'all like

parts' that are interchangeable

without adjustment

Desirable

34. 2. Hardware -

General The ATS should allow for

removal of major components

(such as computers and monitors)

by using common tools

Desirable

35. 2. Hardware -


should be designed to have a

single master power control and

surge suppressor which will allow

the Examiner to remove power

from each individual testing

station when desired

Desirable

36. 2. Hardware -


should be designed to secure

against unauthorized removal

from the testing location (secured

against theft of entire unit)

Desirable


Require-



Rank

37. 2. Hardware -

General Listening devices should be

designed to prevent unsanitary

conditions (i.e., handsets do not

invade the ear and are made of

materials that do not readily

sustain or spread germs)

Desirable

38. 3. Software -

General Provide all application software

necessary for implementation of

the ATS.

Mandatory

39. 3. Software -

General The ATS interface must be

designed to allow use by persons

who are not considered computer

experts, and all software

functionality must be menu driven

or transparent to the user and

function without significant user

intervention

The operation of these

functions must be presented in

an intuitive and simple-to-use

format appropriate to the skill

level of a casual computer user.

Mandatory

40. 3. Software -

General The ATS must be capable of

simultaneously displaying text and

picture(s)

Mandatory

41. 3. Software -

General The ATS must have the ability to

set the following testing options:

Terminating a test after a preset

number of questions have been

answered correctly or incorrectly

(e.g., “quick pass” or “quick fail”)

Requiring the customer to take the

entire test regardless of whether

the customer passes or fails

Mandatory

42. 3. Software -

General The ATS functions must provide

the ability to add, edit, or remove

testing stations

Mandatory

43. 3. Software –

General The ATS is capable of video

presentations in association with

questions and answers (question

text is not to interfere with video

presentations)

Due to issues with bandwidth,

centralized systems (e.g., web

applications) may not be able

to use video.

Desirable

44. 4. System and

System

Integration

The ATS must provide for the risk

of DMV systems being

unavailable by disconnection or

system failure

Examples might include

network failure or files on DMV

systems becoming unavailable.

Mandatory

BIDDER

MUST

ELABORATE


Require-



Rank

45. 4. System and

System

Integration

The ATS must store individual

test result data with the DMV

Personal ID (PID) Number - a

unique identifier which can

contain up to a 8

digit/alphanumeric characters

Mandatory

46. 4. System and

System

Integration

Customer information will, at a

minimum, include:

DMV Customer Number

Last Name

First Name

Date of Birth

Desirable

47. 5. Data Storage

and Reports ATS must record and maintain

data and generate Summary

reports from this data for all tests

taken for a period of at least 6

years.

Reports of summary nature to be

created from among fields listed

below.

DMV PID Number

Customer name

Age, age group, gender

ID of person who entered test

results

Test site

Counter Number

Count of tests by Test type

Test type

Report period

Test date/time

Elapsed time

Score

Decision (pass/fail/abandon)

Test Method (e.g., written, oral,

audio, translator)

Language

Reports to be created by

administrator i.e. the

administrator should be

able to choose which

fields to include in a

report.

Should be able to set

fields as parameters in

report construction.

Combinations of numeric

fields should be capable

of being totaled and/or

averaged: ex: if pass and

fail are chosen to be

displayed on report, total

and/or average of these

should be possible.

Report columns should

be capable of grand

totals.

Reports to be constructed

from these fields are:

Test Results Summary

This data shall be

available to

Administrators and

Examiners

Mandatory


Require-



Rank

48. 5. Data Storage


data and generate Itemized and/or

Detailed reports from this data for

all tests taken for a period of at

least 6 years.

DMV PID Number

Customer name



results

Test site

Test station

Counter Number


Test type

Report period

Test date/ time

Elapsed time

Answers Correct

Answers Incorrect

Score



audio, translator)

Language

Reports to be created by

administrator i.e. the

administrator should be

able to choose which

fields to include in a

report.

Shall be able to set fields

as parameters in report

construction.

Combinations of numeric

fields should be capable

of being totaled and/or

averaged: ex: if pass and

fail are chosen to be

displayed on report, total

and/or average of these

should be possible.

Report columns should

be capable of grand

totals.

Reports to be constructed

from these fields are:

Test Results Detailed,

and/or Itemized

This data shall be

available to

Administrators and

Examiners

Mandatory


Require-



Rank

49. 5. Data Storage


data and generate Cancel Reason

reports from this data for all tests

taken for a period of at least 6

years.

DMV PID Number

Customer name



results

Test site

Test station

Counter Number


Test type

Report period

Test date/ time

Elapsed time

Question answered

Question unanswered

Cancel reason


audio, translator)

Language

This data shall be

available to

Administrators and

Examiners

Mandatory


Require-



Rank

50. 5. Data Storage


data and generate Question

Analysis reports from this data

for all tests taken for a period of at

least 6 years.

Test type

Domain

Language


audio, translator)

Question number

Visual reference

Audio reference

Question usage

Question Average time

Question source (reference)

Selections: totals and % of:

Answer

Distracters

Not Answered

Skipped

This data shall be

available to

Administrators and

Examiners

Mandatory

51. 5. Data Storage

and Reports The ATS must provide and

maintain an audit trail of any

changes made to the question and

answer database and shall include,

at a minimum, the following data:

Administrator ID

Administrator Name

Date

Full text of the question prior to

changes

Full text of question, complete

with changes

Full text of answer prior to

changes

Full text of answer, complete with

changes

Visual prior to changes

Visual after change

This data shall be

available to

Administrators and

Examiners

Mandatory

52. 5. Data Storage

and Reports The ATS must have the ability to

purge old data.

Mandatory


Require-



Rank

53. 5. Data Storage

and Reports The ATS must have the capability

to produce and print a customer

receipt that at a minimum,

includes:

Title / identifier (DMV logo and

receipt title)

DMV PID Number

Customer name

Test site

Test date

Test type

Test station

Test date/ time


audio, translator)

Score


Study material reference (manual

name and page number)

This data shall be

available to

Administrators and

Examiners

Mandatory

54. 5. Data Storage

and Reports The ATS must have the capability

to produce a review, for a

minimum of six years, of a test

that at a minimum, includes:

An exact replication of the test

as it was presented to the

customer (e.g., photo, question,

answer, options in same order)

Indicator of answer selected by

customer and indicator of

correct answer.

This data shall be

available to

Administrators and

Examiners

Mandatory

55. 10. Data

Storage and

Reports

The ATS must archive all test

history data for a minimum of 6

years on enterprise data storage

systems.

Mandatory

56. 5. Data Storage

and Reports Reports and data that are available

at testing sites, must also be

available centrally (e.g., from the

Administrator access point).

Mandatory


Require-



Rank

57. 5. Data Storage

and Reports The ATS should have the

capability to create statistical and

management reports by DMV

office location and by statewide

totals, including but not limited to

the following information.

The pass/fail rates for each test (by

method and language).

The pass/fail rates for each

question

The pass/fail rates for each type of

test (Class C, Motorcycle, etc.)

The total tests conducted by:

Type (e.g. Class C, motorcycle,

etc.)

Method (e.g. normal ATS, audio,

written)

Language (e.g. English, Spanish)

This data shall be

available to

Administrators and

Examiners

Mandatory

58. 5. Data Storage

and Reports Test data shall be available

immediately to examiners at all

branch offices and to the

administrator.

Mandatory

59. 6. Backup and

Recovery If a power or network outage

occurs, Consoles and test stations:

Settings must return to their

settings in place prior to the

outage.

Tests shall resume at the exact

question displayed prior to the

outage, within a reasonable time.

All data regarding previous tests

and tests in progress shall be

restored to its pre-outage state.

Mandatory

60. 6. Backup and

Recovery If a power outage or system failure

occurs, all regularly required

statistical data regarding previous

tests and tests in progress must be

restored and continue to function

as if no interruption had occurred.

Mandatory

61. 6. Backup and

Recovery The ATS must provide the ability

to continue to perform testing in

situations where the regularly used

Examiner console/access point

fails.

Mandatory


Require-



Rank

62. 6. Backup and

Recovery The ATS must be compatible with

the Agency enterprise backup

solution.

Agency backup software is

BackupExec 2010

Mandatory

63. 6. Backup and

Recovery The ATS must have the capability

to restore ATS data and settings

stored in Agency enterprise

backup solution

Mandatory

64. 6. Backup and

Recovery The ATS should be capable of

restarting all tests within five (5)

minutes of restoration of power in

the event of power failure of the

ATS or individual units

Desirable

65. 7. Security The ATS must meet Agency

Information Security requirements

Mandatory

66. 7. Security The ATS must require the entry of

an authorized login ID and

password before starting up

(authorization will be supplied

centrally by DMV Active

Directory)

Mandatory

67. 7. Security The ATS must allow all

passwords to be centrally managed

by DMV systems using Active

Directory

Mandatory

68. 7. Security Access to the system and to

system administration functions

must have a minimum of three

security levels which shall

include:

Access to Examiner functions for

routine testing operations

Access to Examiner functions for

system start-up, set-up, or

modification

Access to system administrative

functions and test creation and

modification

Mandatory

69. 8. Testing

Interface The ATS testing station must

display the customer's name and

contain a text field that may be

edited by the Administrator

The text field may contain

information such as

instructions or warnings.

Mandatory


Require-



Rank

70. 8. Testing


display instructions on the use of

the testing station followed by at

least one (1) practice/sample

question that will not affect his or

her test score.

Mandatory

71. 8. Testing


display prompts and/or guiding

information to assist the customer

in completing tests.

At a minimum, the system must

include the following:

A prompt advising the customer

on how to answer test questions

during the sample/instructional

test questions

Instructions on how to complete

the current step and/or proceed to

the next step in the testing process

A confirming prompt each time a

customer chooses an answer,

Answers will not be accepted until

the customer confirms.

An End-of-Test prompt instructing

customer that the test is over and

where to go for further processing

Mandatory

72. 8. Testing

Interface The customer must have the

capability to change answer

selections any number of times

prior to submitting a final answer

for scoring

Mandatory

73. 8. Testing

Interface The customer must have the

capability to skip a question and

continue with the test.

Mandatory

74. 8. Testing


require the customer to answer

skipped questions before the test is

completed.

(unless ATS ends the test before

this is possible, e.g., if the test is

set to end once a set number of

questions are answered correctly)

Mandatory


Require-



Rank

75. 8. Testing

Interface ATS testing station must display

test results to the customer once a

test session is complete.

Results data must, at a minimum,

include:

Test type

Total questions answered

Number of correct answers

Score

Decision (Pass or Fail status)

Next-step instruction (e.g., "return

to Examiner's station")

Mandatory

76. 9. Examiner

Functions The Examiner must have the

capability to manually enter

customer data for manually

initiated tests.

Data must include:

Customer Number

Last Name

First Name

DOB

Mandatory

77. 9. Examiner


capability to delete entries from

the testing queue.

Mandatory

78. 9. Examiner


capability to assign a customer to

a specific testing station.

Mandatory

79. 9. Examiner


capability to assign a customer the

option to take one or more tests

(of more than one type) during

testing sessions.

Mandatory

80. 9. Examiner


capability to select/revise testing

options which must, at a

minimum, include:

Test method

Language

Audio option

Translator option (designates that

a translator will be used)

Mandatory


Require-



Rank

81. 9. Examiner

Functions The ATS must provide a

continually updated display of the

status and test progress for each

testing station.

The data displayed must, at a

minimum, include the following:

Customer Identifier (must include

name)

Elapsed Time

Number of questions answered

correctly


incorrectly

Other tests assigned but not started

Other tests assigned and already

completed with:


correctly;


incorrectly; and

Pass/Fail indicator

Mandatory

82. 9. Examiner


capability to cancel any test in

progress at any time.

Mandatory

83. 9. Examiner


capability to retrieve and review

completed test results for an

adjustable time period of a

minimum 2 years.

Mandatory

84. 9. Examiner


capability to print a written test for

individual applicants.

Mandatory

85. 9. Examiner


capability to grade test results for

written tests.

Mandatory

86. 9. Examiner


capability to enter test results for

written tests.

Mandatory

87. 9. Examiner

Functions ATS must have an automated

grading solution.

Mandatory


Require-



Rank

88. 9. Examiner

Functions When the Examiner enters a

Customer Number for a previously

tested customer, the ATS must

automatically auto-populate

customer data fields.

Mandatory

89. 9. Examiner


capability to pause tests and then

resume paused tests.

Mandatory

90. 9. Examiner

Functions The Examiner should have the

capability to perform all daily test-

site administrative functions in ten

minutes or less (e.g., start-up,

shutdown, and standard

maintenance).

Desirable

91. 9. Examiner


capability of selecting a translator

from a list of previously entered

translators and associating that

translator with tests translated.

Data should include:

Translator Name

Translator Driver License Number

We do not store translator data

at this time; However, it will be

beneficial for us to do so.

Desirable

92. 9. Examiner

Functions ATS translator data to be stored in

a centralized location to be shared

by all testing locations for

automated functions and data

retrieval.

Desirable

93. 9. Examiner

Functions The ATS should have the

capability to automatically stop

tests if the assigned translator has

been marked as invalid for

providing translation services.

Desirable

94. 9. Examiner


ability to set the following testing

options:

Terminating a test after a preset

number of questions have been

answered correctly or incorrectly

(e.g., “quick pass” or “quick fail”)

Requiring the customer to take the

entire test regardless of whether

the customer passes or fails

This need is also listed under

the 'Software - General'

section. It is listed there as a

'Mandatory' to make sure the

option is available. It is listed

here as an Examiner function

as a 'Desirable' to signify DMV

preference as to where control

of this function should reside.

Desirable


Require-



Rank

95. 9. Examiner

Functions When accessing completed test

results, the Examiners in any

office should be able to access test

results for any office.

Desirable

96. 10.

Administrator

Functions

The Administrator must have the

capability to perform all of the

functions available to the

Examiner.

Mandatory

97. 10.

Administrator

Functions


capability to perform all functions

from a centralized location.

The central location is currently

DMV Central Office.

Mandatory

98. 10.

Administrator

Functions

For all test methods, the

Administrator must have the

capability to add, delete, and

modify test types and cite sources.

Mandatory

99. 10.

Administrator

Functions




modify test questions and answers

and cite sources.

Mandatory

100. 10.

Administrator

Functions




modify test categories.

Mandatory

101. 10.

Administrator

Functions



capability to add or modify the

passing grade for each test (by

either percentage or number of

questions answered correctly).

Mandatory

102. 10.

Administrator

Functions




modify pictures associated with a

question and cite sources.

Mandatory


Require-



Rank

103. 10.

Administrator

Functions




modify the effective start and end

dates for questions.

Example: Effective start date

makes a question available to a

test if the current date is equal

or greater than.

Effective end date removes a

question from a test if the

current date is equal or greater

than.

Mandatory

104. 10.

Administrator

Functions


capability to designate that

specific questions are always

included on particular test types.

Mandatory

105. 10.

Administrator

Functions


capability to review any

completed Test Template.

Mandatory

106. 10.

Administrator

Functions


capability to access a central store

for test and statistical data from all

DMV offices.

Mandatory

107. 10.

Administrator

Functions


capability to distribute/update test

templates (complete with suite of

questions/answers) to all testing

stations from a central location

and cite sources.

The central location is currently

DMV headquarters.

Mandatory

108. 11.

Administrator

Functions


capability to add, remove, or

update text references to the

Driver License Manual without

needing to upgrade hardware or

software.

Mandatory


Require-



Rank

109. 10.

Administrator

Functions

ATS must provide reports to

Administrator to verify that

centralized system functions were

performed correctly.

Data must include but is not

limited to:

Success status of updates to all

stations

Installed test versions

Discrepancies by station

Test results upload

This is notification after an

actual event such as

uploads/updates.

Information to be reported

would be architecture

dependent.

Mandatory

110. 10.

Administrator

Functions


capability to set general ATS

System-wide settings.

Examples include:

Add/edit/remove testing sites

Add/edit/remove test stations

Add/edit/remove network

addresses/locations

Format and style preferences

Mandatory

111. 10.

Administrator

Functions


capability to edit test session

instructions which are shown as

part of the test on the testing

station.

Mandatory

112. 10.

Administrator

Functions


capability to assign questions to

categories and set weightings,

randomizations, and other

calculations used to generate final

tests from the test template.

Mandatory

113. 10.

Administrator

Functions

ATS must provide to

Administrator, reports and

analysis data for test questions in

electronic and printed form.

Examples of data include:

Number of times each answer

was selected

Identification of which answer

was correct

How much time was spent on

each question

Mandatory


Require-



Rank

114. 10.

Administrator

Functions

The Administrator should have the

capability to export ATS statistical

information to a different file

format.

DMV preference for

Excel (.xls) format.

Desirable

115. 10.

Administrator

Functions

ATS should provide status data to

Administrator that identifies those

system components are available,

working correctly, and that

software versions are

synchronized throughout the

system.

Desirable

116. 10.

Administrator

Functions

The Administrator should have the

capability to mark a translator's

record as valid or invalid for

providing translation services.

Desirable

117. 11. Test

Creation and

Functionality

Character size and font must meet

American Disability Act (ADA)

requirements.

(Character size and font type to

be approved by DMV).

Mandatory

118. 11. Test

Creation and

Functionality

At implementation, ATS must

provide direct replacements for the

current test types which include:

Basic knowledge on operation

of a private automobile (Class

D )

Safe Driving Practices

Motorcycle

CDL General Knowledge

CDL Combination Vehicle

CDL Air Brake

CDL Passenger

CDL Double/Triple

CDL Tank vehicle

CDL Hazardous material

School Bus TYPE 1

School Bus TYPE 2

Mandatory


Require-



Rank

119. 11. Test

Creation and

Functionality

At implementation, ATS must

provide full audio-visual and text

support for the following

languages:

English

French

Spanish

Serbo-Croatian (Bosnian)

Somali (Mai Mai)

Current language media (audio

and text need to be replaced /

updated). Use of computer

generated languages, such as

Easy Reader, is desired.

Mandatory

120. 11. Test

Creation and

Functionality

The ATS must allow for

expansion of available languages

for all test types (CDL excluded)

and test methods.

CDL – English only Mandatory

121. 11. Test

Creation and

Functionality

Tests must be available in the

following presentation modes:

Visually on the Testing station

monitors

Orally (audio) through a

listening device attached to the

testing station (in conjunction

with the Visual test)

Written (printed) tests produced

on the printer

Mandatory

122. 11. Test

Creation and

Functionality

Each question must have up to

four multiple choice answers (with

one correct answer and the others

as distracters).

Mandatory

123. 11. Test

Creation and

Functionality

Test length must be adjustable, to

be determined by DMV, and will

vary among test types.

Mandatory

124. 11. Test

Creation and

Functionality

The ATS must store more

questions (a.k.a., question bank)

for each test than is asked during

an examination.

Mandatory

125. 11. Test

Creation and

Functionality

The ATS must have the capability

of providing at least 300 multiple

choice questions in each language

for each test template.

This is the bank of questions

that each test selects from.

Mandatory


Require-



Rank

126. 11. Test

Creation and

Functionality

The question bank must include

the questions, answers, and

associated audio recordings,

visuals, and videos. Visuals and

videos must be of Vermont scenes,

and properly licensed copyrighted

material, or in public domain.

We are anticipating the

Contractor will provide the

pictures & videos

Mandatory

127. 11. Test

Creation and

Functionality

The entire question bank must be

available at all locations for real-

time generation of tests.

Mandatory

128. 11. Test

Creation and

Functionality

The ATS must be capable of

generating multiple tests without

repeating all questions on those

tests.

Question selection, from the

bank, must be randomized

Mandatory

129. 11. Test

Creation and

Functionality

The questions sequence must be

randomized for each test.

Mandatory

130. 11. Test

Creation and

Functionality

The default must be that the

system randomizes the order in

which answers will display for a

specific question.

Mandatory

131. 11. Test

Creation and

Functionality

The ATS must provide an override

to the random feature to allow the

Administrator to designate

specific sequences for answers.

Mandatory

132. 11. Test

Creation and

Functionality

The ATS must allow for additions

and modifications to test

questions, answers, and additions

and deletions of pictures without

upgrades to hardware and/or

software.

Mandatory

133. 11. Test

Creation and

Functionality

Each question field must allow for

a minimum of 400 characters.

Mandatory

134. 11. Test

Creation and

Functionality

Each answer (and distracter) field

must allow for a maximum of 256

characters.

(e.g., when there are four

answer options, the total

number of characters for all

four answers will be at least 4 x

256).

Mandatory

135. 11. Test

Creation and

Functionality

The ATS testing station must

present pictures, questions, and

answer options in a consistent

location on the testing station

monitor.

Mandatory


Require-



Rank

136. 11. Test

Creation and

Functionality

Picture files need to be a

compressed file format.

Mandatory

137. 11. Test

Creation and

Functionality


and deletions of audio

questions/answers without

upgrades to hardware or software

Mandatory

138. 11. Test

Creation and

Functionality


and deletions of video files

without upgrades to hardware or

software.

This requirement is only

mandatory if the final solution

has video capability. Assumes

this is an option with current

bandwidth.

Mandatory

See

comments

field for

proviso

139. 11. Test

Creation and

Functionality

Pictures need to be at least 20%

but not larger than 25% of the size

of the screen.

Desirable

140. 12. Audio Test

Specific

Functionality

Audio tests must be available on

all testing stations.

Mandatory

141. 12. Audio Test

Specific

Functionality

The audio test must include a

volume control feature to enable

customers taking an audio test to

change volume to a comfortable

listening level.

Mandatory

142. 12. Audio Test

Specific

Functionality

Volume must be automatically

reset to a prescribed default level

at the conclusion of each test.

Mandatory

143. 12. Audio Test

Specific

Functionality

The audio test must provide the

test question and all answer

choices.

Mandatory

144. 12. Audio Test

Specific

Functionality

The audio test must permit the

customer to repeat the current

question and answer audio

recordings until an answer is

selected.

Mandatory

145. 12. Audio Test

Specific

Functionality

The audio test must be available in

all languages in which visual tests

are available.

Mandatory

146. 12. Audio Test

Specific

Functionality

Bidder should have the ability to

provide the audio recordings in all

languages that are used by the

ATS.

Desirable


Require-



Rank

147. 13. Written Test

Specific

Functionality

The ATS must have the capability

to print written tests for all

languages and test types identified

as required for the System.

Mandatory


Specific

Functionality

ATS must provide functionality

that will be used to grade written

tests (e.g., answer key).

Mandatory


Specific

Functionality

The written test must correspond

'word-for-word' with the visual

tests displayed on the testing

stations.

Mandatory


Specific

Functionality

The written test printing must be:

Letter quality

On 8 1/2” x 11” paper

Compliant with American

Disability Act (ADA)

requirements for letter/symbol

sizes

Complete with all questions,

answers, and any associated

pictures

Generated using the same

randomizing process used to

present the tests on the testing

stations

Able to create and print

multiple tests from a single

request by the Examiner (e.g.,

print several CDL tests and

each is generated and

randomized individually)

Mandatory


Specific

Functionality

The ATS should have a automated

grading option for written tests

(e.g., scanner and grading

software)

Desirable

152. 14. Training Bidder must be able to provide

qualified trainers to ensure a

successful installation, pilot, roll-

out, and implementation of the

ATS.

See section 3.1.12. Mandatory

153. 14. Training Bidder must be able to provide

training to designated DMV

employees.

Mandatory


Require-



Rank

154. 14. Training Bidder must develop and provide

training curricula.

Mandatory

155. 14. Training User training must include 'hands-

on' training at all locations

Mandatory

156. 14. Training Training must be available in a

minimum of three levels:

The basic course (Level 1) that

the majority of employees will

receive

The advanced course (Level 2)

for managers and supervisors

IT/ Administrator specific

training (Level 3)

Mandatory

157. 14. Training Level 1 training for designated

Field Office personnel must

include the following subjects:

The use of the system to assign

tests in all formats, print tests,

retrieve test results, review

tests, and cancel tests

How to score tests

Adding / removing customers

from the queue

Starting the test for a queued

customer

The use of Examiner functions

The general operation functions

of the system (start-up and end-

of-day procedures)

Simple trouble-shooting

techniques

Special features to activate and

de-activate the system

Creation/printing of reports (for

use when Managers are on

leave)

Mandatory


Require-



Rank

158. 14. Training Level 2 training for designated

DMV Headquarters and Field

Office managers and supervisors

must include the following

subjects:

Training identical to Level 1

training

The use of Examiner functions

to retrieve testing statistics

System security (summary of

features)

Creation/printing of reports

Use of features that can be site

specific

Mandatory

159. 14. Training Level 3 training for DMV

Headquarters employees including

IT personnel must include the

following subjects:

Training identical to Levels 1

and 2 training

The use of the Administrator

functions to create, modify, and

delete test questions and

answers

The use of Administrator

functions to make test questions

available at all testing stations


functions to retrieve testing

statistics and create reports


functions to perform remote

administration functions

How to use and install all ATS

equipment and software (if

applicable)

Advanced trouble-shooting

techniques to detect

hardware/software problems at

the Administrator level

System security

Error detection and triage

Mandatory


Require-



Rank

160. 14. Training Provide a minimum of one

training manual at each location

for each function set (e.g.,

Administrator, Examiner) that will

be performed at that location.

Mandatory

161. 14. Training Provide a minimum of one

examiner’s user manual for each

system provided in field offices.

Mandatory

162. 14. Training Provide an Examiner's manual at

each location which must, at a

minimum, include:

An introduction to the system

A detailed explanation of

software and each hardware

component

Start-up, operating and end-of-

day procedures

Trouble shooting guidelines and

maintenance instructions

System and component

operating instructions

A list of all components,

including name, description

Mandatory

163. 14. Training Provide a troubleshooting guide(s)

at each location which will include

a listing of:

Error messages

Failures

Probable causes

Suggested fixes

Mandatory

164. 14. Training Provide an Administrator's manual

which must, at a minimum,

include:

Administrator level

troubleshooting guide

Reference material that has

detailed information on all

system functionality and all

subject matter covered in

training sessions

Mandatory

165. 14. Training Provide all manuals also as digital

file.

Microsoft Word document,

Adobe PDF document, etc.

Mandatory


Require-



Rank

166. 14. Training Provide a training representative

who shall be available at the start-

up site.

Mandatory

167. 14. Training Provide all equipment and

supplies necessary to provide

training if different than installed

workstations.

Mandatory

168. 14. Training Training materials must be written

to allow use by persons who are

not considered computer “experts”

- training materials need to be

"user-friendly".

Mandatory

169. 15. Service and

Maintenance ATS maintenance must be

performed at times that will not

impede the regular course of

business in Field Offices.

Schedules and details will be

mutually agreed upon.

Mandatory


Maintenance Provide 3

rd level telephone support

and on-site repair, maintenance

and support for all components

and software provided by the

contractor, including, but not

limited to, detection and

correction of any ATS software

error, within the timeframes

projected or specified by DMV.

DMV provides 1st and 2nd level

support. 1st level support is

provided in the offices. 2nd level

support is provided by DMV

Help Desk. 3rd level support

includes those issues that can't

be resolved by DMV.

Time period and support

details will be negotiated in the

warranty and maintenance

agreement (if applicable).

Normal business hours are 7:45

a.m. to 4:30 p.m. EST Time,

Monday through Friday & 7:45

AM TO 6 PM on Wednesday,

excluding State holidays.

Mandatory


Maintenance Provide a single (toll-free)

telephone number for all support

service.

DMV assumes that, at a

minimum, voicemail will be

available after normal DMV

work hours.

Mandatory


Require-



Rank


Maintenance Respond, during DMV business

hours, by telephone (if DMV has

left a phone message) to the Help

Desk within two hours or less of

receiving the call.

DMV assumes that messages

left after normal DMV business

hours will be responded to by

8am of the next business day

(EST).

Mandatory


Maintenance Provide a warranty for Bidder-

provided software and hardware.

Warranty periods begin after

the system has been fully

implemented and accepted by

DMV. The minimum warranty

period is 1 year on software

and 1 year on hardware.

Mandatory


Maintenance Provide a maintenance agreement

for Bidder-provided software and

hardware.

Mandatory


Maintenance Provide a guarantee that all

hardware acquired under the

contract shall have long-term

product support (availability of

hardware maintenance service and

replacement parts) for a minimum

of three years following the date

the Bidder provides written

notification to DMV that the

hardware is out of production.

Desirable


Require-



Rank


Maintenance

Provide warranty period options,

such as, 1, 2, or 3 years which

includes:

Software patches

Software upgrades

Other components provided by Contractor

Bidder does not need to supply

more than one timeframe

option. However, the warranty

option(s) will be included in

scoring of the proposal.

Warranty items listed are

assumed to be at no additional

cost above the cost of the

warranty unless otherwise

negotiated.

Costs for repairs due to

damage caused outside of

normal 'wear and tear' (e.g.,

vandalism) will be identified in

the warranty agreement.

Do not include cost information

for warranty options in this

section of the proposal; all cost

information is to be included in

the cost proposal.

Desirable


Require-



Rank


Maintenance Provide maintenance agreement

period options, such as 1, 3, 5, or

10 years which includes:

Software patches

Software upgrades

Other components provided by contractor

Bidder does not need to supply more than one timeframe option. However, the maintenance agreement option(s) will be included in scoring of the proposal.

Parts, labor, software patch,

and software upgrade costs will

be identified in the

Maintenance agreement.

Costs for repairs due to

damage caused outside of

normal 'wear and tear' (e.g.,

vandalism) will be identified in

the maintenance agreement.

Do not include cost information

for service and maintenance

agreement options in this

section of the proposal; all cost

information is to be included in

the cost proposal

Desirable

3.14 Value Added Features:

Detail and describe any functionality, products, services, or updates that are not part of the requirements in this RFP. Demonstrate added value that you have to offer that is related to this request.

3.15 Professional Services Requirements:

The following services are expected for this project:

Project Management

Analysis and Design – to fully understand the needs of DMV

Development – to tailor the application to our requirements if needed.


Data Conversion – to convert/port existing questions, answers, audio, video and history files to the new system.

Testing

Training

Implementation

Documentation

Post implementation support

3.16 Project Management

Propose a project management plan outlining the tasks, number of hours for each task, and duration. Detail the tasks you expect to be completed by DMV staff, and your expectations. Once a project plan is agreed to, the Contractor will be expected to manage the project, working with a DMV project manager.

The Contractor shall follow project management methodologies that are consistent with the Project Management Institute’s (PMI) Project Management Body of Knowledge (PMBOK) Guide. Contractor’s staff and subcontractors are to follow a consistent methodology for all SOW activities.

The methodology must be approved by the Department of Information and Innovation’s Enterprise Project Management Office (EPMO) as they have statutory oversight authority for information technology activities over $100k. A list of EPMO’s project management minimum deliverables can be found at http://dii.vermont.gov/pm/pmtemplates#Paragraph6 along with a description of the EPMO project management standard processes.

3.17 Staffing

Contractor shall obtain approval in advance by the DMV of all staff proposed for this project. Contractor shall notify the DMV in advance and obtain approval of any new staff if staffing changes during a project. If any Contractor staff does not perform up to acceptable or satisfactory standards as documented in the contract or SOW, the DMV shall inform the Contractor. When so informed, the Contractor shall either replace the staff member(s) with approved staff or take remedial action as agreed to by the DMV to ensure staff are performing at an acceptable standard. Bidders should review Section 203.02 PERSONNEL REQUIREMENTS AND CONDITIONS of the Procedures for Selecting Contractors (Attachment C) for other requirements and conditions.

3.18 Data Security:

DMV systems contain Personally Identifiable Information (PII) about its customers. This information must be

treated according to the DMV Data Security Policy (Attachment E), and the Driver Privacy Protection Act

(Attachment G).

3.19 Assumptions:

Submit a list of any constraints or assumptions you have regarding this RFP which impact your proposal. Changes

to these assumptions may require updates to scheduling or hour estimates.

3.20 Performance Measures

In accordance with current State of Vermont policy and procedures, the contract may include Contractor

performance measures. The specific performance measures will be determined during the contract negotiation

http://www.sec.state.vt.us/tutor/dobiz/forms/fcregist.htm#Paragraph6


process. These measures may include the possibility of both positive and negative incentives. Bidders are

encouraged to participate in the development of performance measures to be utilized.

4 Cost Proposal:

The Cost proposal must be submitted in a separate package from the technical proposal. Submit four (4) copies for review. Remember: the proposal must be fixed cost, inclusive of expenses, for specific deliverables.

4.1 Non-recurring: Include a list of one time charges related to the system/software/hardware or engagement.

4.2 Recurring: Include a list all recurring costs. Examples include annual maintenance costs, licensing fees that are paid on an annual basis, etc.

4.3 Price Guarantee: The bidders are required to maintain their price for a period of 120 days from submission while evaluations and contract negotiations are underway. Prices incorporated into any resulting contract will be locked in for the term of the contract.

4.4 Maintenance Agreement: Cost proposal must include one year of maintenance costs, which will begin on the day the system is initially accepted and placed into production. Include maintenance costs for years 2 – 5.

4.5 New Releases: If applicable, provide the projected upgrade schedule of the core application, and what is included in the cost of the maintenance/support contract.


5 Bidder Response Content and Format

The State discourages overly lengthy and costly proposals; however, in order for the State to evaluate proposals

fairly and completely, proposers must follow the format set out in this RFP and provide all information

requested. Bidders must adhere to the requirements listed below to be considered responsive:

5.1 The bid should include a Transmittal Letter, a Technical Response, a separate Cost Proposal, and a separate sealed package containing financial information

5.2 Technical Proposal - Number of Copies: Submit an unbound original (clearly marked as such), seven (7) bound paper copies, and one (1) CD-ROM copy. Please number the pages consecutively.

5.3 The separate Cost Proposal must include an original and four (4) paper copies and one (1) CD-ROM copy.

5.4 Only one copy of the financial information is required.

5.5 Bidders must provide a comprehensive response to the Requirements specified in Section 3. Responses must be in the same sequence as they appear in Section 3. Bidders must provide supporting narrative and documentation when required in response to the requirements of this section.

5.6 Acknowledgment of Terms

A statement from the Bidder, and its legal counsel, certifying their acceptance of the Procedures for Selecting

Contractors and Specifications for Contractor Services, and the Standard State Provisions for Contracts and Grants

with any exceptions or additional provisions noted. (These will be considered when making an award).

5.7 Experience and Qualifications:

Bidder must describe the experience of their firm in completing similar projects. Additionally, bidder must

provide information specific to the personnel assigned to accomplish the work called for in this RFP. Bidder must

provide a narrative description of the organization of the project team and a personnel roster that identifies each

person who will actually work on the contract and provide the title, resume, and description of the type of work

the individual will perform for each person listed.

5.7.1 If a bidder intends to use subcontractors, the bidder must identify in the proposal the names of the subcontractors and the portions of the work the subcontractors will perform.

Bidder shall include information as described in Section 5.7 above for each subcontractor

identified.


5.7.2 Bidders should provide three (3) references from clients who have been through successful implementations, (of similar size and scope) and are running the specific solution being proposed. For each reference, provide a point of contact: name; phone number; and email addresses as applicable. References should be able to provide feedback specific to the bidder’s performance on their project. The DMV reserves the right to contact any references provided by the bidder.

5.7.3 Provide a brief description of “lessons learned” from the projects completed, as referenced in Section 5.7.2 above, as well as from any relevant projects that were not successful.

5.7.4 DMV reserves the right to request and consider the opinions of any local, State and/or Federal Agency relative to the qualifications, capability and performance of any firms and/or sub-contractor identified in the proposal.

5.8 Financial Requirements:

5.8.1 In order to be considered responsive to this RFP, each consultant shall conform to the following requirements.

Submit one (1) envelope or package containing:

A completed VTrans Form AF38 (revised 06/2011) for the Bidder and each subcontractor

named in the proposal. For this submittal the Bidder shall complete Sections One and

Two.

For each subcontractor, the VTrans Form AF38 (revised 06/2011) shall be

completed at the level commensurate with the amount of work anticipated to be

performed by the subcontractor.

For Section Two of the VTrans for AF38, submit financial statements only. No overhead

schedule is required.

The financial information for the Consultant and each sub-consultant shall be current for

each firm and cover a period ending no more than 18 months prior to the date the

proposal is due.

5.8.2 The proposer shall provide financial information in such a manner that the State can reasonably formulate a determination about the stability and financial strength of the organization. This must include but not be limited to company size, organization, date of incorporation, ownership, number of employees, revenues for the last fiscal year, and, if available, audited financial statements for the most recent 3 years. A current Dun and Bradstreet Report that includes a financial analysis of the firm would fulfill this requirement. A proposer can use an Annual Report as verification of financial status provided it contains at a minimum a Compiled Income Statement and Balance Sheet verified by a Certified Public Accounting firm. The State reserves the right to contact the accounting firm if questions arise. As an alternative, for those proposers unable to provide audited financial statements or Dun


and Bradstreet report, the proposer shall provide tax returns and financial statements including income statements and balance sheets for the most recent 3 years, and any available credit reports.

5.8.3 Disclose any and all judgments, pending or expected litigation, or other real potential financial reversals, which might materially affect the viability or stability of the proposer’s organization; or certify that no such condition is known to exist.

5.8.4 A confidentiality statement may be included if this portion is considered non-public information. The State may request reports on financial stability from independent financial rating services in order to further substantiate stability.

5.9 Certificate of Compliance: This form must be completed and submitted as part of the response for the proposal to be considered valid.

5.10 Workers’ Compensation; State Contracts Compliance Requirement; Self Reporting: This form must be completed and submitted as part of the response for the proposal to be considered valid.

5.11 Workers’ Compensation; State Contracts Compliance Requirement; Subcontractor Reporting: This form must be completed and submitted as part of the response for the proposal to be considered valid.

5.12 Offshore Outsourcing Questionnaire: This form must be completed and submitted as part of the proposal to be considered valid.

6 Submission Instructions and Checklist:

6.1 Closing Date

The closing date for the receipt of proposals is February 15, 2013 at 1PM. Bid must be delivered to:

Vermont Purchasing and Contract Administration Division, Office of Purchasing & Contracting

10 Baldwin St,

Montpelier VT 05633-7501

prior to 1PM. Proposals or unsolicited amendments submitted after that time will not be accepted and will

be returned to the bidder.

The bid opening will be held at Office of Purchasing & Contracting 10 Baldwin St, Montpelier VT05633-

7501 and is open to the public.


6.2 Submission Checklist

6.2.1 Technical proposal Hard Copies (7) 6.2.2 Technical proposal Original Unbound Master 6.2.3 Transmittal Letter 6.2.4 Technical Proposal CD ( 6.2.5 References – (Reference 5.7.2) 6.2.6 Acceptance of Attachment C -The Standard State Provisions for Contracts and

Grants ; Procedures For Selecting Contractors And Specifications For Contractor Services, Including Customary State Contract Provisions dated August 28, 2008, Revised December 29, 2008; and the General Special Provisions dated November 22, 2011

6.2.7 Offshore/outsource form – (Reference Attachment B) 6.2.8 Certificate of Compliance – (Reference Attachment A) 6.2.9 License & Maintenance Agreements – (Reference 2.20 if applicable) 6.2.10 Workers’ Compensation; State Contracts Compliance Requirement; Self Reporting 6.2.11 Workers’ Compensation; State Contracts Compliance Requirement; Subcontractor

Reporting 6.2.12 Cost Proposal Hard Copies (4) 6.2.13 Financial Information Hard Copy (1)

6.3 Sealed Bid Instructions

6.3.1 All bids must be sealed and must be addressed to the State of Vermont, Purchasing and Contract Administration Division, Office of Purchasing & Contracting, 10 Baldwin St,, Montpelier, VT 05633-7601. BID ENVELOPES MUST BE CLEARLY MARKED ‘SEALED BID’ AND SHOW THE REQUISITION NUMBER AND/OR PROPOSAL TITLE, OPENING DATE AND NAME OF BIDDER.

6.3.2 All bidders are hereby notified that sealed bids must be in the office of the Vermont Purchasing and Contract Administration Division by the time of the bid opening. Bids not in possession of the Purchasing and Contract Administration Division at the time of the bid opening will not be considered, and returned to the bidder.

6.3.3 Purchasing and Contract Administration Division may, for cause, change the date and/or time of bid openings or issue an addendum. If a change is made, the State will make a reasonable effort to inform all bidders by posting at: http://bgs.vermont.gov/purchasing/bids.

6.3.4 All bids will be publicly opened. The name and address of each bidder will be announced.

6.4 Delivery Methods:

6.4.1 U.S. MAIL: Bidders are cautioned that it is their responsibility to originate the mailing of bids in sufficient time to ensure bids are received and time stamped by the Office of Purchasing & Contracting prior to the time of the bid opening.

6.4.2 EXPRESS DELIVERY: If bids are being sent via an express delivery service, be certain that the RFP designation is clearly shown on the outside of the delivery



envelope or box. Express delivery packages will not be considered received by the State until the express delivery package has been received and time stamped by the Office of Purchasing & Contracting.

6.4.3 HAND DELIVERY: Hand carried bids shall be delivered to a representative of the Division prior to the bid opening.

6.4.4 ELECTRONIC: Electronic bids will not be accepted.

6.4.5 FAX BIDS: FAXED bids will not be accepted.


7 Attachments:

Attachment A: Certificate of Compliance

Attachment B: Offshore Outsourcing Questionnaire

Attachment C: The Procedures for Selecting Contractors and Specifications for Contractor Services document

including Customary State Contract Provisions, dated August 28, 2008, Revised December 29,

2008; the General Special Provisions dated November 22, 2011; the Standard State Provisions for

Contracts and Grants (January 8, 2009); and the revised Standard State Provisions for Contracts

and Grants, dated November 7, 2012;

Attachment D: Commodity Purchases Terms and Conditions (if applicable)

Attachment E: DMV Data Security Policy

Attachment F: VTrans Infrastructure Overview

Attachment G: Driver Privacy Protection Act (DPPA)

Attachment H: Workers’ Compensation; State Contracts Compliance Requirement; Self Reporting and Workers’

Compensation; State Contracts Compliance Requirement; Subcontractor Reporting

Attachment I: American Association of Motor Vehicle Administrators (AAMVA) Guidelines for Knowledge and

Skills Testing

Attachment J: Sample Confidentiality & Non-Disclosure Agreement

Attachment K: VAOT Form AF-38

REMEMBER!

ALL NOTIFICATIONS, RELEASES AND AMENDMENTS WILL BE POSTED AT:


THE STATE WILL MAKE NO ATTEMPT TO CONTACT BIDDERS WITH UPDATED INFORMATION. IT WILL BE THE

RESPONSIBILITY OF EACH BIDDER TO PERIODICALLY CHECK THIS SITE FOR THE LATEST DETAILS.

http://dii.vermont.gov/Policy_Central


Attachment A: Certificate of Compliance

RFP/PROJECT: Driver License Automated Testing System

DATE:

This form must be completed in its entirety and submitted as part of the response for the proposal to be considered valid.

Taxes: Pursuant to 32 V.S.A. § 3113, bidder hereby certifies, under the pains and penalties of perjury, that the

company/individual is in good standing with respect to, or in full compliance with a plan to pay, any and all taxes

due to the State of Vermont as of the date this statement is made. A person is in good standing if no taxes are

due, if the liability for any tax that may be due is on appeal, or if the person is in compliance with a payment plan

approved by the Commissioner of Taxes.

Insurance: Bidder certifies that the company/individual is in compliance with, or is prepared to comply with, the

insurance requirements as detailed in Section 7 of Attachment C: Standard State Contract Provisions. Certificates

of insurance must be provided prior to issuance of a contract and/or purchase order. If the certificate(s) of

insurance are not received by the Office of Purchasing & Contracting within five (5) days of notification of award,

the State of Vermont reserves the right to select another vendor. Please reference the RFP and/or RFQ # when

submitting the certificate of insurance.

Contract Terms: The undersigned hereby acknowledges and agrees to Attachment C: The Procedures for Selecting

Contractors and Specifications for Contractor Services document including Customary State Contract Provisions,

dated August 28, 2008, Revised December 29, 2008; the General Special Provisions dated November 22, 2011; the

Standard State Provisions for Contracts and Grants (January 8, 2009); and the revised Standard State Provisions for

Contracts and Grants, dated July 1, 2012;.

Terms of Sale: The undersigned agrees to furnish the products or services listed at the prices quoted. The Terms

of Sales are Net 30 days from receipt of service or invoice, whichever is later. Percentage discounts may be

offered for prompt payments of invoices; however such discounts will not be considered in making awards.

Form of Payment: Would you accept the Visa Purchasing Card as a form of payment? __ Yes __ No

Insurance Certificate(s): Attached _______will provide upon notification of award _________


Delivery Offered: _______ days after notice of award Terms of Sale: ___________________

(If Discount)

Quotation Valid for: _____ days Date: __________

Name of Company: ______________________ Contact Name: ___________________________

Address: ________________________________ Fax Number: _______________________

_________________________________________ E-mail: ______________________________

By: ______________________________________ Name: _______________________________

Signature (Bid Not Valid Unless Signed) (Type or Print)

All returned quotes and related documents must be identified with our request for quote number.


Attachment B: Offshore Outsourcing Questionnaire

Vendors must indicate whether or not any services are or will be outsourced under the terms of any agreement with the State of Vermont. Indicate N/A if

not applicable. This is required by the State of Vermont but cannot be used as an evaluation criterion under Federal Law.

Services:

Proposed Service to be

Outsourced

Bid Total or

Contract Estimate

Represents what % of

total Contract Dollars

Outsourced Dollars

Outsourced Work

Location (Country)

Subcontractor

If any or all of the services are or will be outsourced offshore, Vendors are required to provide a cost estimate of what the cost would be to provide the

same services onshore and/or in Vermont.

Bid Total or Contract

Estimate if provided

Bid Total or Contract

Estimate if provided in


Proposed Service to be

Outsourced

Onshore Vermont

Cost Impact

Onshore Work Location

Subcontractor

Name of Bidder: Signature of Bidder: Date:


Attachment C: Procedures for Selecting Contractors and Specifications for

Contractor Services including Customary State Contract Provisions

(http://www.aot.state.vt.us/conadmin/Documents/ProceduresForSelectingCon

tractorsandSpecificationsforContractorServices.pdf)




Attachment C: The General Special Provisions

(http://www.aot.state.vt.us/conadmin/Documents/ConsSpecProvNovember201

1.pdf)




ATTACHMENT C: STANDARD STATE PROVISIONS

FOR CONTRACTS AND GRANTS

NOVEMBER 7, 2012

1. Entire Agreement: This Agreement, whether in the form of a Contract, State Funded Grant,

or Federally Funded Grant, represents the entire agreement between the parties on the subject

matter. All prior agreements, representations, statements, negotiations, and understandings

shall have no effect.

2. Applicable Law: This Agreement will be governed by the laws of the State of Vermont.

3. Definitions: For purposes of this Attachment, “Party” shall mean the Contractor, Grantee or

Subrecipient, with whom the State of Vermont is executing this Agreement and consistent

with the form of the Agreement.

4. Appropriations: If this Agreement extends into more than one fiscal year of the State (July

1 to June 30), and if appropriations are insufficient to support this Agreement, the State may

cancel at the end of the fiscal year, or otherwise upon the expiration of existing appropriation

authority. In the case that this Agreement is a Grant that is funded in whole or in part by

federal funds, and in the event federal funds become unavailable or reduced, the State may

suspend or cancel this Grant immediately, and the State shall have no obligation to pay

Subrecipient from State revenues.

5. No Employee Benefits For Party: The Party understands that the State will not provide any

individual retirement benefits, group life insurance, group health and dental insurance,

vacation or sick leave, workers compensation or other benefits or services available to State

employees, nor will the state withhold any state or federal taxes except as required under

applicable tax laws, which shall be determined in advance of execution of the Agreement.

The Party understands that all tax returns required by the Internal Revenue Code and the

State of Vermont, including but not limited to income, withholding, sales and use, and rooms

and meals, must be filed by the Party, and information as to Agreement income will be

provided by the State of Vermont to the Internal Revenue Service and the Vermont

Department of Taxes.

6. Independence, Liability: The Party will act in an independent capacity and not as officers or

employees of the State.

The Party shall defend the State and its officers and employees against all claims or suits

arising in whole or in part from any act or omission of the Party or of any agent of the Party.

The State shall notify the Party in the event of any such claim or suit, and the Party shall

immediately retain counsel and otherwise provide a complete defense against the entire claim

or suit.

After a final judgment or settlement the Party may request recoupment of specific defense

costs and may file suit in Washington Superior Court requesting recoupment. The Party shall


be entitled to recoup costs only upon a showing that such costs were entirely unrelated to the

defense of any claim arising from an act or omission of the Party.

The Party shall indemnify the State and its officers and employees in the event that the State,

its officers or employees become legally obligated to pay any damages or losses arising from

any act or omission of the Party.

7. Insurance: Before commencing work on this Agreement the Party must provide certificates

of insurance to show that the following minimum coverages are in effect. It is the

responsibility of the Party to maintain current certificates of insurance on file with the state

through the term of the Agreement. No warranty is made that the coverages and limits listed

herein are adequate to cover and protect the interests of the Party for the Party’s operations.

These are solely minimums that have been established to protect the interests of the State.

Workers Compensation: With respect to all operations performed, the Party shall carry

workers’ compensation insurance in accordance with the laws of the State of Vermont.

General Liability and Property Damage: With respect to all operations performed under

the contract, the Party shall carry general liability insurance having all major divisions of

coverage including, but not limited to:

Premises - Operations

Products and Completed Operations

Personal Injury Liability

Contractual Liability

The policy shall be on an occurrence form and limits shall not be less than:

$1,000,000 Per Occurrence

$1,000,000 General Aggregate

$1,000,000 Products/Completed Operations Aggregate

$ 50,000 Fire/ Legal/Liability

Party shall name the State of Vermont and its officers and employees as additional insureds

for liability arising out of this Agreement.

Automotive Liability: The Party shall carry automotive liability insurance covering all

motor vehicles, including hired and non-owned coverage, used in connection with the

Agreement. Limits of coverage shall not be less than: $1,000,000 combined single limit.

Party shall name the State of Vermont and its officers and employees as additional insureds

for liability arising out of this Agreement.

8. Reliance by the State on Representations: All payments by the State under this Agreement

will be made in reliance upon the accuracy of all prior representations by the Party, including


but not limited to bills, invoices, progress reports and other proofs of work.

9. Requirement to Have a Single Audit: In the case that this Agreement is a Grant that is

funded in whole or in part by federal funds, the Subrecipient will complete the Subrecipient

Annual Report annually within 45 days after its fiscal year end, informing the State of

Vermont whether or not a single audit is required for the prior fiscal year. If a single audit is

required, the Subrecipient will submit a copy of the audit report to the granting Party within 9

months. If a single audit is not required, only the Subrecipient Annual Report is required.

A single audit is required if the subrecipient expends $500,000 or more in federal assistance

during its fiscal year and must be conducted in accordance with OMB Circular A-133. The

Subrecipient Annual Report is required to be submitted within 45 days, whether or not a

single audit is required.

10. Records Available for Audit: The Party will maintain all books, documents, payroll papers,

accounting records and other evidence pertaining to costs incurred under this agreement and

make them available at reasonable times during the period of the Agreement and for three

years thereafter for inspection by any authorized representatives of the State or Federal

Government. If any litigation, claim, or audit is started before the expiration of the three year

period, the records shall be retained until all litigation, claims or audit findings involving the

records have been resolved. The State, by any authorized representative, shall have the right

at all reasonable times to inspect or otherwise evaluate the work performed or being

performed under this Agreement.

11. Fair Employment Practices and Americans with Disabilities Act: Party agrees to comply

with the requirement of Title 21V.S.A. Chapter 5, Subchapter 6, relating to fair employment

practices, to the full extent applicable. Party shall also ensure, to the full extent required by

the Americans with Disabilities Act of 1990, as amended, that qualified individuals with

disabilities receive equitable access to the services, programs, and activities provided by the

Party under this Agreement. Party further agrees to include this provision in all subcontracts.

12. Set Off: The State may set off any sums which the Party owes the State against any sums due

the Party under this Agreement; provided, however, that any set off of amounts due the State

of Vermont as taxes shall be in accordance with the procedures more specifically provided

hereinafter.

13. Taxes Due to the State:

a. Party understands and acknowledges responsibility, if applicable, for compliance with

State tax laws, including income tax withholding for employees performing services

within the State, payment of use tax on property used within the State, corporate

and/or personal income tax on income earned within the State.

b. Party certifies under the pains and penalties of perjury that, as of the date the

Agreement is signed, the Party is in good standing with respect to, or in full

compliance with, a plan to pay any and all taxes due the State of Vermont.

c. Party understands that final payment under this Agreement may be withheld if the

Commissioner of Taxes determines that the Party is not in good standing with respect


to or in full compliance with a plan to pay any and all taxes due to the State of

Vermont.

d. Party also understands the State may set off taxes (and related penalties, interest and

fees) due to the State of Vermont, but only if the Party has failed to make an appeal

within the time allowed by law, or an appeal has been taken and finally determined

and the Party has no further legal recourse to contest the amounts due.

14. Child Support: (Applicable if the Party is a natural person, not a corporation or partnership.)

Party states that, as of the date the Agreement is signed, he/she:

a. is not under any obligation to pay child support; or

b. is under such an obligation and is in good standing with respect to that obligation; or

c. has agreed to a payment plan with the Vermont Office of Child Support Services and

is in full compliance with that plan.

Party makes this statement with regard to support owed to any and all children residing in

Vermont. In addition, if the Party is a resident of Vermont, Party makes this statement with

regard to support owed to any and all children residing in any other state or territory of the

United States.

15. Sub-Agreements: Party shall not assign, subcontract or subgrant the performance of his

Agreement or any portion thereof to any other Party without the prior written approval of the

State. Party also agrees to include in all subcontract or subgrant agreements a tax certification

in accordance with paragraph 13 above.

16. No Gifts or Gratuities: Party shall not give title or possession of any thing of substantial

value (including property, currency, travel and/or education programs) to any officer or

employee of the State during the term of this Agreement.

17. Copies: All written reports prepared under this Agreement will be printed using both sides of

the paper.

18. Certification Regarding Debarment: Party certifies under pains and penalties of perjury

that, as of the date that this Agreement is signed, neither Party nor Party’s principals

(officers, directors, owners, or partners) are presently debarred, suspended, proposed for

debarment, declared ineligible or excluded from participation in federal programs, or

programs supported in whole or in part by federal funds.

Party further certifies under pains and penalties of perjury that, as of the date that this

Agreement is signed, Party is not presently debarred, suspended, nor named on the State’s

debarment list at: http://bgs.vermont.gov/purchasing/debarment

19. Certification Regarding Use of State Funds: In the case that Party is an employer and this

Agreement is a State Funded Grant in excess of $1,001, Party certifies that none of these

State funds will be used to interfere with or restrain the exercise of Party’s employee’s rights

with respect to unionization.

(End of Standard Provisions)

http://bgs.vermont.gov/purchasing/debarment


Attachment D: Commodity Purchases Terms and Conditions

1. Statement of Rights: The State of Vermont reserves the right to obtain clarification or additional

information necessary to properly evaluate a proposal. Vendors may be asked to give a verbal

presentation of their proposal after submission. Failure of vendor to respond to a request for additional

information or clarification could result in rejection of that vendor's proposal. To secure a project that is

deemed to be in the best interest of the State, the

State reserves the right to accept or reject any and all bids, in whole or in part, with or without cause,

and to waive technicalities in submissions. The State also reserves the right to make purchases outside

of the awarded contracts where it is deemed in the best interest of the State.

2. Responses: Responses must be submitted on, or in accordance with, forms or format requirements

provided by the Office of Purchasing & Contracting. Prices and all other information entered on the

quote, except signature of vendor, should be typed or printed for legibility. ALL SUBMISSIONS MUST BE

SIGNED.

3. Prices: Unless otherwise stated, prices are net and no charge for packing, shipping, or for any other

purpose will be allowed over and above the price quoted. Prices quoted for printing are to include

printing, binding, wrapping, and packaging. All prices are delivered F.O.B. destination, unless otherwise

stated.

4. Taxes: Most state purchases are not subject to federal or state sales or excise taxes and must be

invoiced tax free. An exemption certificate will be furnished upon request covering taxable items. The

contractor agrees to pay all Vermont taxes which may be due as a result of this order. If taxes are to be

applied to the purchase it will be so noted in the response.

5. Order of Precedence: The order of precedence for documentation will be the State of Vermont

Standard Contract Form and attachments, the bid document and any amendments, and the vendor’s

response and any amendments.

6. Substitution: Unless otherwise stated, vendors may offer substitutes to items identified by a

manufacturer’s number or brand. When offering a substitution, vendor must describe any differences


and provide technical information that will assist in the evaluation. After an award is made, substitutions

are not acceptable unless authorized in writing by the Office of Purchasing & Contracting.

7. Specification Change: Any changes or variations in the specifications must be received in writing from

the Office of Purchasing & Contracting. Verbal instructions or written instructions from any other source

are not to be considered.

8. Method of Award: Awards will be made under the provisions of 29 V.S.A. § 903. The State may award

one or more contracts and reserves the right to make additional awards to other compliant bidders at

any time during the first year of the contract if such award is deemed to be in the best interest of the

State. All other considerations being equal, preference will be given to resident bidders of the State

and/or to products raised or manufactured in the State.

9. Default: In case of default of the contractor, the State may procure the materials or supplies from

other sources and hold the contractor responsible for any excess cost occasioned thereby, provided,

that if public necessity requires the use of materials or supplies not conforming to the specifications

they may be accepted and payment therefore shall be made at a proper reduction in price.

10. Cancellation: The State specifically reserves the right to cancel the contract or any portion thereof

providing, in the opinion of its Commissioner of Buildings and General Services, the services or materials

supplied by the contractor are not satisfactory or are not consistent with the terms of the contract.

11. Delivery: Liability for product delivery remains with the contractor until the product is properly

delivered and signed for in accordance with the Office of Purchasing & Contracting terms and

conditions. Shipments shall be securely and properly packed, according to accepted commercial

practices, without extra charge for packing cases or other containers. Such containers will remain the

property of the State unless otherwise stated. Delivered goods that do not conform to the specifications

or are not in good condition upon receipt shall be replaced promptly by the contractor.

12. Invoicing: All invoices are to be rendered by the Contractor on the vendor's standard bill-head and

forwarded directly to the institution or agency ordering materials or services.


13. Non Collusion: The State of Vermont is conscious of and concerned about collusion. It should

therefore be understood by all that in signing bid and contract documents they agree that the prices

quoted have been arrived at without collusion and that no prior information concerning these prices has

been received from or given to a competitive company. If there is sufficient evidence to warrant

investigation of the bid/contract process by the Office of the Attorney General, all bidders should

understand that this paragraph might be used as a basis for litigation.

14. Amendments: No changes, modifications, or amendments in the terms and conditions of this

contract shall be effective unless reduced to writing, numbered, and signed by the duly authorized

representative of the State and Contractor.

15. Confidentiality: The successful response will become part of the contract file and will become a

matter of public record as will all other responses received. If the response includes material that is

considered by the bidder to be proprietary and confidential under 1 VSA, Chapter 5, the bidder shall

clearly designate the material as such, explaining why such material should be considered confidential.

The bidder must identify each page or section of the response that it believes is proprietary and

confidential with sufficient grounds to justify each exemption from release, including the prospective

harm to the competitive position of the bidder if the identified material were to be released. Under no

circumstances can the entire response or price information be marked confidential. Responses so

marked may not be considered.

16. Certification for apparel, footwear, and textiles (sweatshop prohibition): Before commencing work

on this contract, the contractor must provide certification from each supplier that meets the

requirements of 29 V.S.A. §922(a) as well as a list of the names and addresses of each supplier, as

required by 29 V.S.A. §922(b). Contractor certifies that if, at any time during the contract period, there

are changes to the information in the certification or to the list of suppliers the contractor will promptly

inform the Commissioner of Buildings and General Services of such changes.


Attachment E: DMV Data Security Policy

State of Vermont

DMV Data Security Policy

Release

Info Version Author Date

Pre-

production

release V.99

Jack

Green

- AOT 10/16/2008

Production

Release V 1.0

Jack

Green

- AOT 10/24/2008

Date: October 17, 2008

Approved by: Signed version on file

Policy Number:


Contents

1.0 Introduction .................................................................................................................................. 77

1.1 Authority ......................................................................................................................... 77

1.2 Purpose ........................................................................................................................... 77

1.3 Scope ............................................................................................................................... 77

1.4 Background ..................................................................................................................... 77

2.0 Policy ............................................................................................................................................. 77

2.1 Personally Identifiable Information (PII Data) Defined .................................................. 77

2.2 DMV must, by law, share these data .............................................................................. 78

2.3 Technical Guidance on Data Transferred to Networked External Partners ................... 78

2.4 Technical Guidance on Data Stored or Presented by the Department of Motor Vehicles............................................................................................................................................... 79

2.5 Non-Compliance ............................................................................................................. 80

3. Signatory Page ............................................................................................................................... 81


8 1.0 Introduction

8.1.1 1.1 Authority

The State of Vermont is authorized to undertake the development of enterprise architecture policies

and standards. The Department of Information and Innovation (DII) was created in 22 V.S.A. § 901 (1),

“to provide direction and oversight for all activities directly related to information technology, including

telecommunications services, information technology equipment, software, accessibility, and networks

in state government”

8.1.2 1.2 Purpose

The purpose of these policies is to identify the processes used to protect personally identifiable

information (PII) stored in databases on the Department of Motor Vehicles database servers. It defines

what these data are, defines external and State of Vermont network attack vectors and describes

countermeasures to protect our PII data.

8.1.3 1.3 Scope

These policies apply to data defined as personally identifiable by 9 V.S.A. § 2430(5). It covers the storage

of these data as well as their transmission over private and public networks.

8.1.4 1.4 Background

As our society has become increasingly dependent on information systems, the risks associated with

more sophisticated attacks to gain access to sensitive data has equally increased. The response to these

attacks has been to institute a wide variety of deterrents designed to keep assailants at bay. As a result,

attackers have started to use other methods to attain sensitive data. If data are compromised, this

exposes the organization and, more importantly, our citizens to potential negative consequences,

including regulatory fines (e.g., HIPPA), punitive awards and loss of credibility with employees, partners

and citizens, to name a few. Therefore, it is imperative that all State agencies follow a policy to ensure

the protection of sensitive data both inside and outside the organization.

9 2.0 Policy

9.1.1 2.1 Personally Identifiable Information (PII Data) Defined

As stated in 9 V.S.A. § 2430(5)(A), An Act Relating to the Protection of Personal Information, 1 defines

Personally Identifiable Information as

1 http://www.leg.state.vt.us/docs/legdoc.cfm?URL=/docs/2006/acts/ACT162.HTM

http://www.leg.state.vt.us/docs/legdoc.cfm?URL=/docs/2006/acts/ACT162.HTM


(5)(A) “Personal information” means an individual’s first name or first initial and last name in

combination with any one or more of the following data elements, when either the name or the data

elements are not encrypted or redacted or protected by another method that renders them unreadable

or unusable by unauthorized persons:

(i) Social Security number;

(ii) Motor vehicle operator’s license number or non-driver identification card number;

(iii) Financial account number or credit or debit card number, if circumstances exist in which the

number could be used without additional identifying information, access codes, or passwords;

(iv) Account passwords or personal identification numbers or other access codes for a financial

account.

9.1.2 2.2 DMV must, by law, share these data

These and other data are shared among different State agencies to aid in enforcing child support laws,

to establish tax roles, to identify conviction or suspension status and others.

Additionally, the DMV enters into contractual relationships with external agencies sharing information

such as insurance status or license renewal

The necessity of exchanging this information requires that we carefully plan how these data are

exchanged.

9.1.3 2.3 Technical Guidance on Data Transferred to Networked External Partners

These policies apply to all data that leave the AOT/DMV campus:

All PII data leaving the State of Vermont Network (159.105/16) shall be encrypted with a

minimum of 256 bit SSL encryption.

All data conduits shall be protected by an ingress IP to IP firewall rule

DMV will be the requestor of a data transfer, preventing the need for an opened port on the

firewall unless otherwise specified.

Non-PII data leave the network encrypted with a minimum of 256 bit SSL encryption unless

otherwise specified.


All data leaving the network via web services shall be SSL encrypted and shall be presented to

specific clients proxied from the DMZ

9.1.4 2.4 Technical Guidance on Data Stored or Presented by the Department of Motor Vehicles

These policies apply to all PII data stored on PII database servers:

All PII data shall be stored in an encrypted, redacted, scrambled or segmented format.

All data exports that contain PII data shall be authorized by the DMV Commissioner or her/his

designee and presented only by Enterprise Database staff members. Any other export sets

exposed by other staff are considered unauthorized.

All data exports leaving DMV secured areas, must be stored on encrypted media according to

the DII policy Data Encryption for Laptop and Tablet Computers 2. Storing PII data on

unencrypted media such as DVD’s, CD’s or USB memory sticks is prohibited.

The access of database records containing PII, not directly from the application, shall be

recorded including, but not limited to, UID, date, time and the specific records selected.

All constituent servers shall be placed in physically secure server rooms.

All databases will be backed up regularly. Any tapes taken off site must be stored in physically

secured rooms. All backups stored on disk must be stored in restricted access folders.

These policies apply to all PII data stored as reports, either paper or electronic:

Unless authorized end users have participated in a DPPA agreement, All PII data shall be

presented in an encrypted, redacted, scrambled, fully blacked-out, segmented format; or shall

be couriered to a State storage facility; or shall be stored in a physically secured facility.

These policies apply to users authorized to view PII data:

All internal users authorized to view PII data shall participate in Driver Protection Policy Act

(DPPA) training.

All external users authorized to view PII data shall agree to the constraints defined in the

Agreement as to the Driver Privacy Protection Policy, State of Vermont, Department of Motor

Vehicles 3

2 http://dii.vermont.gov/sites/dii/files/pdfs/DII-Data_Encryption_for_Laptops_Policy.pdf

http://dii.vermont.gov/sites/dii/files/pdfs/DII-Data_Encryption_for_Laptops_Policy.pdf


9.1.5 2.5 Non-Compliance

Non-compliance must be reported immediately to the department supervisor. Each agency is

responsible for establishing and including the appropriate response(s) within their agency. Any agency

not adhering to these policies may potentially expose itself to legal ramifications and regulatory fines to

include possible punitive damages.

3 http://www.aot.state.vt.us/dmv/documents/TA/Vg/Tavg118.pdf

http://www.aot.state.vt.us/dmv/documents/TA/Vg/Tavg118.pdf


10 3. Signatory Page

Kris Rowley, System Security Director

Tom Hurd, Chief Information Officer, VTRANS

Tom Murray, Chief Information Officer, State of Vermont


Attachment F: VTrans Infrastructure Overview

Network:

The Vermont Agency of Transportation (VTrans) has multiple local area networks (LANs/VLANs)

implemented at its’ 3 main locations in central Vermont as well as LANs at 20+ remote offices. LAN

equipment is implemented using Cisco switches. LAN access port speed is up to 1Gb with core switch

speeds up to 10Gb. The wide area network (WAN) that connects VTrans remote offices is supported by

the State’s Department of Information and Innovation (DII). DII has contracted multiple vendors to

provide WAN interconnect at varying speeds, 1Mb – 10Mb, depending on the location’s requirements.

VTrans utilizes hardware firewalls at of our ingress/egress of DII’s WAN. VTrans has also implemented or

has plans to implement its own fiber optic network between its 3 main locations in central Vermont and

its disaster recovery site to provide 1-10Gb speeds between these locations.

Active Directory:

All of Vtrans user, desktop and server security/authentication is implemented using MicroSoft’s

Windows 2003 Active Directory services. VTrans currently has 3 Active Directories. Two have been

implemented in our Internet/Web facing DMZs and our main Active Directory for approximately 1100

users/desktops and 200+ physical and virtual servers.

Virtual Servers:

VTrans’ current virtual server environment consists of 100+ Micro Soft Windows 2003/2008 quest

servers. These guest servers are implemented on VMware ESXi v4.1 hosts, managed with two vCenter

v4.1 servers. The physical infrastructure supporting VTrans’ virtual environment consists of HP blade

servers in multiple c-class enclosures with shared storage being presented from an HP P4000/LeftHand

iSCSI SAN configured Raid 10. This SAN is being backed up to replicated HP 4312 D2D StoreOnce backup

appliances. VTrans’ plans and preference is to virtualize as many of its existing servers as feasible as well

as any future server needs.

Backup:

VTrans data and server backup policy is a full monthly offsite backup (kept for a year), a full weekly

onsite backup (kept for a month) and a daily increment/differential (kept for a week). The VTrans

backup software is BackupExec 2010.


Patching:

Each month VTrans’ approves and pushes out Micro Soft’s critical and security patches for all of our

currently implemented server and desktop OS’s, Office Suite(s) and Internet Explorer Versions. This

patching is accomplished using Micro Soft’s WSUS and Active Directory Group Policies (GPO). The GPO

policy’s download and installation/reboot schedules are adjusted for uptime requirements and to

reduce interruptions.

Virus Scanning:

McAfee Virus Scan Enterprise software for virus scanning and clients are managed with central ePolicy

Orchestrator server. Updates are pulled into the master repository every 6 hours from the McAfee site

and clients get updated on a daily basis.

VTrans’ Network Security

The approach for the design and implementation of VTrans’ Network is to identify network traffic flow,

minimize that flow and limit it to only required access, thereby limiting the exposure of VTrans’

networked resources and data to corruption, non-approved access and/or improper use.

VTrans’ Five Network Security Zones:

The overall network design for VTrans is based on five security zones. Ingress/egress for each Zone will

be controlled and limited as required to meet VTrans’ business needs while providing secured access

control.

The five security zones:

Zone 1 - External DMZ

Zone 2 - Internal Client

Zone 3 - Internal Server/Services

Zone 4 - Internal Data Storage

Zone 5 – Infrastructure Management

Zone 1 – External DMZ:

All of VTrans’ Internet facing servers / services will reside in the ‘VTrans’ External DMZ Security Zone’

and only VTrans’ servers/services will reside in this zone. Network traffic flow for this zone will be

to/from the Internet and as required to/from the State’s private network for data presentation and

server/services management. VTrans’ IT staff will work closely with the State’s Department of

Information and Innovation (Dii) on the design, implementation, maintenance, support and security of


this zone. VTrans’ IT Staff will provide required documentation and security plans for all server/services

that reside in this security zone. All servers and equipment in this zone will be in an access controlled

location and/or lockable enclosure.

Zone 2 – Internal Client:

All of VTrans’ networked client devices such as PCs, laptops, tablets and printers will reside in ‘VTrans’

Internal Client Security Zone’. This zone is the most widely used and is distributed to multiple locations

throughout the State. Network traffic flow for this zone will be to/from VTrans’ security zones 1, 2, 3 as

well as to other State resources and the Internet. The WAN traffic for these locations utilizes Dii’s WAN

infrastructure as well as VTrans’ own Infrastructure where available. The ingress/egress to Dii’s WAN

infrastructure for all VTrans’ locations in this zone will be controlled by a hardware firewall. VTrans’ IT

Staff is responsible for the design, implementation, maintenance, support and security VTrans’

networked client devices in this zone.

Zone 3 – Internal Server/Services:

All of VTrans’ internal file, application, database and infrastructure servers will reside in the ‘VTrans’

Internal Server/Services Security Zone’ and only VTrans’ servers will reside in this zone. Network traffic

flow to/from this zone will be controlled at VTrans’ core switch(s) and hardware firewall(s). VTrans’ IT

Staff will be responsible for the design, implementation, maintenance, support and security for this

zone. VTrans’ IT Staff will document and develop security plans for all servers/services that reside in this

security zone. All servers and equipment in this zone will be in an access controlled location and/or

lockable enclosure.

Zone 4 – Internal Data Storage:

VTrans’ storage area networks (SAN), network attached storage (NAS) and backup server(s)/devices will

reside in the ‘VTrans’ Internal Data Security Zone’ and only VTrans’ SANs, NASs and backup devices will

reside in this zone. Network traffic flow to/from this zone will be will be controlled at VTrans’ core

switch(s) and hardware firewall(s). VTrans’ IT Staff will be responsible for the design, implementation,

maintenance, support and security for this zone. VTrans’ IT Staff will document and develop security

plans for all network access to/from this security zone. All equipment in this zone will be in an access

controlled location and/or lockable enclosure.

Zone 5 – Infrastructure Management:

Only approved equipment and designated staff will have access to the ‘VTrans’ Infrastructure

Management Security Zone’. This zone is to be used for the ongoing support of all VTrans’ security

zones. Network traffic flow to/from this zone will be allowed to all other VTRANS security zones. VTrans’

IT Staff will document and develop security plans for all access to/from this security zone. Due to the

high level of access, access will require a written ‘needs’ approval of VTrans’ IT Infrastructure Manager.

All equipment in this zone will be in an access controlled location and/or lockable enclosure.


PC operating system – Windows 7

Database – MS SQL 2008 cluster with fail over


ATTACHMENT G: Driver Privacy Protection Policy

CATEGORY: Records

POLICY DESCRIPTION: Driver Privacy Protection Policy

STATUTE REFERENCE: DPPA Federal Requirement

EFFECTIVE: 09/13/97 Revised:

STATEMENT OF PURPOSE; The purpose of this policy is to ensure that the State of Vermont, Department of Motor Vehicles is in substantial compliance with the Driver Privacy Protection Act of 1994. The policy has been developed to protect an individual's rights to privacy to the extent possible while allowing access to targeted information for legitimate purposes.

I. Definitions:

"Motor Vehicle Records" shall mean any record that pertains to a motor vehicle operator's permit, driver's license, motor vehicle title, motor vehicle registration, or identification card issued by the Department of Motor Vehicles.

"Person" shall mean an individual, organization or entity, but does not include this State or any Agency thereof.

"Named Person" shall mean that, depending upon the type of information sought, the person making the request has furnished either the name and date of birth of an individual or the registration plate number or VIN of a vehicle.

"Personal Information" shall mean information in a motor vehicle record that identifies an individual, including an individual's photograph or computerized image, social security number, driver identification number, name, address other than the five-digit zip code, telephone number, and medical or disability information, but does not include information on vehicle accidents, driving violations and driver's license and registration status.


"Disclosure" shall mean releasing information from Department records requested under either 1 V.S.A. Chapter 5, Subchapter 3, 23 V.S.A. Chapter 3 or the Driver Privacy Protection Policy, as applicable. Disclosure shall be subject to Department system and resources capabilities and established costs.

ll. Prohibition on Release and Use of Certain Personal Information from Motor Vehicle Records:

Except as specifically provided for, the Department of Motor Vehicles (the "Department") and any officer, employee or contractor thereof shall not knowingly disclose or otherwise make available to any person, personal information about any individual obtained by the Department in connection with a motor vehicle record.

lll. Release and Use of Certain Personal Information from Motor Vehicle Records:

A. Persons requesting personal information shall properly identify themselves to the Department and shall make all requests for personal information under oath in accordance with 23 V.S.A. Sec. 201 and on a form or in a manner satisfactory to the Department.

B. Unless personal information is confidential or exempt from disclosure under some other provisions of law, personal information in a motor vehicle record shall be disclosed for legitimate purposes as follows:

1. Use in connection with matters of motor vehicle or driver safety and theft, motor vehicle emissions, motor vehicle product alterations, recalls, or advisories, performance monitoring of motor vehicles, motor vehicle parts and dealers by motor vehicle manufacturers and removal of non-owner records from the original owner of the records of motor vehicle manufacturers to carry out the purpose of the Automobile Information Disclosure Act, the Motor Vehicle Information and Cost Saving Act, the National Traffic and Motor Vehicle Safety Act of 1966, the Anti-Car Theft Act of 1992, and the Clean Air Act and may be disclosed for the same uses though not specifically required under the reference Acts.

FURTHER, information may be disclosed for legitimate purposes as follows:


2. Use by any government agency, including any court or law enforcement agency, or any private person or entity acting on behalf of a Federal, State, or local agency in carrying out its official business or in response to any court order.

3. Use by any insurer or insurance support organization authorized to write insurance policies, or by self-insured entities, or their authorized agents, employees or contractors, in connection with claims investigation activities, anti-fraud activities, rating or underwriting.

4. Use by any licensed private investigative agency or licensed security service by this state for any purpose permitted under Section III.

5. Use with respect to a request for named person's motor vehicle records by a legitimate business or its agents, employees, or contractors in the normal course of business, but only:

(a) to verify the accuracy of personal information submitted by the individual to the business or its agents, employees, or contractors; and

(b) if such information as so submitted is not correct or is no longer correct, to obtain the correct information, but only for the purposes of preventing fraud by, pursuing legal remedies against, or recovering on a debt or security interest against the individual.

6. Use in connection with any civil, criminal, administrative, or arbitral proceeding in any Federal, State or local court or agency or before any self-regulatory body, including the service of process, investigation in anticipation of litigation and the execution or enforcement of judgments and orders, or pursuant to any order of a Federal, State, or local court.

7. Use by a legitimate business in research activities, and for use by a legitimate business in producing statistical reports, so long as the personal information is not published, re-disclosed, or used to contact individuals.


8. Use with respect to a request for a named person's motor vehicle records in providing notice to the owners of towed or impounded vehicles.

9. Use with respect to a named person's motor vehicle records by an employer or its agent or insurer to obtain or verify information relating to a holder of a commercial driver's license that is required under the Commercial Motor Vehicle Safety Act of 1986.

10. Use with respect to a request for a named person's motor vehicle records in connection with the operation of private toll transportation facilities.

D) A person shall have access to motor vehicle records relating to such person upon proof of identity. Motor vehicle records relating to a person may be made available to any other person upon proof, in such form and manner as the Department prescribes, that the written consent of the person who is the subject of the record has been obtained.

E) Personal information my be disclosed for any other use specifically authorized under the laws of the State, if such use is related to the operation of a motor vehicle or public safety.

IV. Information pertaining to Vehicular Accidents, Driving Violations and Driver's License and Registration Status:

In the definition of "personal information", there is an express exclusion of information on vehicular accidents, driving violations and driver's license and registration status. Therefore, this information is not considered to be within the scope of the DPPA.

Information on vehicular accidents, driving violations and driver's license and registration status will be made available in accordance with 1 V.S.A. Chapter 5, Subchapter 3 and 23 of V.S.A. Chapter 3.

V. Resale or Re-disclosure:

A recipient of personal information for a particular use may not use the information or make the information available for any use other than for a use permitted under this policy. The recipient may


sell or otherwise make the information available for any use permitted under this policy. A recipient who resells or re-discloses personal information permitted under section III must keep, for a period of 5 years, records identifying each person or entity that received information and the permitted purpose for which the information will be used and must make such records available to the Department of Motor Vehicles upon request.


Attachment: H

Worker’s Compensation; State Contracts Compliance Requirement


DATE:

WORKERS’ COMPENSATION; STATE CONTRACTS COMPLIANCE REQUIREMENT

Self Reporting

Form 1 of 2



2009 and for total projects costs exceeding $250,000.00, requires bidders comply with the following


Bidder is required to self report the following information relating to past violations, convictions,

suspensions, and any other information related to past performance relative to coding and classification

for worker’s compensation. The state is requiring information on any violations that occurred in the

previous 12 months.

Summary of Detailed Information Date of

Notification

Outcome


WORKERS’ COMPENSATION STATE CONTRACTS COMPLIANCE REQUIREMENT: Bidder hereby certifies

that the company/individual is in compliance with the requirements as detailed in Act 54, Section 32 of

the Acts of 2009.

Date:

Name of Company: Contact Name:

Address: Title:

Phone Number:

E-mail: Fax Number:


By: Name:

Signature (Bid Not Valid Unless Signed)* (Type or Print)

*Form must be signed by individual authorized to sign on the bidder’s behalf.



DATE:

WORKERS’ COMPENSATION; STATE CONTRACTS COMPLIANCE REQUIREMENT

Subcontractor Reporting

Form 2 of 2



2009 and for total projects costs exceeding $250,000.00 requires bidders to comply with the following


Bidder is required to provide a list of subcontractors on the job along with lists of subcontractor’s

subcontractors and by whom those subcontractors are insured for workers’ compensation purposes.

Include additional pages if necessary. This is not a requirement for subcontractor’s providing supplies

only and no labor to the overall contract or project.

Subcontractor Insured By Subcontractor’s Sub Insured By


Date:

Name of Company: Contact Name:

Address: Title:

Phone Number:

E-mail: Fax Number:

By: Name:

Signature (Bid Not Valid Unless Signed)* (Type or Print)

*Form must be signed by individual authorized to sign on the bidder’s behalf.

AAMVA GUIDELINES FOR

KNOWLEDGE AND SKILLS TESTING

January 1, 1999

PREFACE These guidelines were prepared by A. James McKnight on behalf of the Driver License Committee "Driver Testing Working Group" of the American Association of Motor Vehicle Administrators. Members of the Committee are:

• Joyce Gelderman, Wisconsin Division of Motor Vehicles • Thomas J. Seery, New York Department of Motor Vehicles • Richard A. Weaver, Florida Department of Motor Vehicles • Carole W. Bedwell, California Department of Motor Vehicles • Douglas K. Tobin, Pennsylvania Bureau of Driver Licensing • Michael Smith, U.S. Department of Transportation • Michael R. Calvin, American Association of Motor Vehicle Administrators • A. James McKnight, National Public Services Research Institute

The Committee is grateful to the many representatives of States and Provinces whose review and comment were highly valuable in guiding the content of the guidelines and the model materials prepared from them. Special thanks are also due Raymond C. Peck of the California Department of Motor Vehicles for his help in preparing the sections on evaluation of tests. These guidelines served as the basis of a Model Driver Manual and Knowledge Test also developed by the Working Group. These model materials were also disseminated among the States and Provinces for review and comment. Results applied to the final version of the manual and test, copies of which were subsequently distributed to AAMVA representatives. While the mission of the working group also called for development of a model road test, the California Department of Motor Vehicles’ newly developed "Driver Performance Evaluation" was so consistent with the guidelines that, with very modification, it was capable of serving as a model test.

TABLE OF CONTENTS PREFACE........................................................................................................................................ i INTRODUCTION ...........................................................................................................................1 GUIDELINES FOR KNOWLEDGE TESTING.............................................................................2

KNOWLEDGE REQUIREMENTS ....................................................................................2 DRIVER MANUAL ............................................................................................................6 KNOWLEDGE TESTS .......................................................................................................7

Test Construction.....................................................................................................7 Content of Knowledge Tests....................................................................................7 Wording of Items .....................................................................................................8 Scoring Standards ....................................................................................................9 Alternate Forms .....................................................................................................10 Item Analysis .........................................................................................................11 Test Administration ...............................................................................................12 Feedback to Applicants..........................................................................................12 Test Security ..........................................................................................................13 Evaluating Knowledge Tests .................................................................................13 Test Reliability.......................................................................................................14 Test Validity ..........................................................................................................15 Effectiveness of Knowledge Testing .....................................................................17 Automated Testing.................................................................................................18

SPECIAL APPLICANT CATEGORIES...........................................................................19 Reading-Limited Applicants..................................................................................19 Providing Information to the Reading-Limited .....................................................19 Testing Reading-Limited Applicants.....................................................................20 Foreign-Speaking Applicants ................................................................................22 Presenting Information to the Foreign-Speaking...................................................22 Testing Foreign-Speaking Applicants ...................................................................23

GUIDELINES FOR SKILL TESTING .........................................................................................24

Skill Requirements.................................................................................................24 Testing Requirements ............................................................................................25 Testing Modes........................................................................................................26

ROAD TEST......................................................................................................................26 Purpose of a Road Test ..........................................................................................27 Road Test Performances ........................................................................................27 Selection of Performances for Observation ...........................................................27 Road Test Maneuvers ............................................................................................28 Road Test Performances ........................................................................................29 Traffic Dependent Performance.............................................................................29

Immediate Test Failures.........................................................................................30 Criteria for Assessing Performance .......................................................................30 Route Selection ......................................................................................................31 Administrative Procedures.....................................................................................32 Test Support ...........................................................................................................33 Test Scoring ...........................................................................................................33 Scoring System ......................................................................................................33 Recording Scores ...................................................................................................34 Evaluating Road Tests ...........................................................................................34 Road Test Reliability .............................................................................................35 Examiner reliability ...............................................................................................35 Route Reliability ....................................................................................................36 Total Reliability .....................................................................................................37 Road Test Validity .................................................................................................37 Effectiveness of Road Tests...................................................................................38

OFF-STREET TESTING...................................................................................................38 Pre-test Screening ..................................................................................................38 Assessing Vehicle Control Skills...........................................................................39 Emergency Skills Testing ......................................................................................39 Evaluating Off-Street Testing................................................................................39 Reliability...............................................................................................................40 Validity ..................................................................................................................40 Effectiveness ..........................................................................................................40

SIMULATION...................................................................................................................41 Interactive Simulation............................................................................................41 Non-Interactive Simulation....................................................................................42 Evaluating Simulation............................................................................................43

INTRODUCTION This report provides guidance in development and evaluation of tests to assess the knowledge and skills of applicants for license to operate automobiles. It has been developed by the American Association of Motor Vehicle Administrators (AAMVA) as part of an attempt to help State licensing agencies achieve uniformly high quality in assessing the ability of driver license applicants to operate vehicles in a manner that assures the safety and mobility of the driving public. The development and evaluation of driver license tests in the past has often been hampered by the inappropriate application of psychometric concepts and techniques. Psychometric testing is largely intended to measure constructs defined by tests themselves and validated in terms of their ability to predict future behavior. Driver license tests, on the other hand, attempt to measure knowledge and skills defined by motor vehicle agencies and are intended to serve as an incentive to applicants to acquire the skills, knowledge and attitudes needed for safe driving and as a means of making sure applicants possess them before they are issued a license. The difference in purpose of tests leads to substantial differences in the way the license tests and psychometric measures are developed and evaluated, differences that will be noted throughout the report. The scope of the guidelines is limited to the initial licensing of drivers in general. It does not address the unique requirements of licensing for special vehicles such as trucks, buses, or motorcycles. Nor, does it attempt to accommodate drivers with specialized needs, such as the physically-disadvantaged, the elderly, frequent traffic offenders, or drinking drivers. Within the general driving population, the guidelines do address those whose language limitations interfere with their ability to acquire, or to demonstrate the possession of, the driving knowledge for which they are held responsible. The body of these guidelines is divided into two sections, corresponding to the two types of abilities to be tested: knowledge and skill. The guidelines for knowledge testing cover both the written test and the driver manual from which test questions are drawn. The manual and test represent two essential elements of the knowledge testing process. Recommendations are also provided for administration of tests through automated processes and the special needs of testing drivers with limited reading ability. Guidelines for skill testing focus upon road tests, as they are the most common means of assessing driving skills within the licensing structure. However, guidelines are also provided for off-street testing and testing with the aid of simulation. Three separate reports provide models of materials developed under the guidelines: a model driver manual, a model knowledge test, and a model road test.

GUIDELINES FOR KNOWLEDGE TESTING This section of the guidelines describes methods for assessing the knowledge of driver license applicants. Research has shown that a license testing program directed at critical knowledge requirements is capable of reducing the likelihood that drivers would be involved in accidents for which they are responsible. The purpose in giving knowledge tests is to assure that driver license applicants possess the information required to operate vehicles in a way that is consistent with the safety and mobility of the public. Providing this assurance means not only assessing applicant knowledge through the written test, but providing a manual or means by which applicants can acquire that knowledge. The two elements of knowledge testing are equally important. These guidelines will address knowledge requirements, the driver manual, and the written test, as well as the special requirements of applicants with language limitations. The scope of the guidelines is confined to testing the knowledge of applicants for a Basic Driver's License. Other guidelines will deal with requirements of heavy vehicles and motorcycles, as well as the special needs of the elderly and the physically disadvantaged. KNOWLEDGE REQUIREMENTS The development of both the driver manual and the written test derive from a common source — the definition of the knowledge required enabling drivers to operate their vehicles in a way that is consistent with the safety and mobility of the motoring public. At one time, the content of driver manuals and knowledge tests was confined to laws and regulations governing motor vehicle operation. This restriction reflected the position that drivers could only be held accountable for knowing what was imposed upon them by law. However, it is now generally accepted that applicants can be held responsible for any knowledge that contribute to the safety and mobility of the public, so long as the necessary information is made available to them through the driver manual or some alternate source. Knowledge requirements include, in addition to laws and regulations, driving procedures, principles, facts, and concepts, including both those that enable drivers to operate their vehicles properly and those that motivate them to do so. A list of knowledge categories appears in the table on the following page. An initial set of knowledge requirements had been identified through earlier research involving a comprehensive analysis of driver tasks and their prioritization in terms of their criticality to traffic safety. This set of knowledge requirements was disseminated among State and Provisional license agencies for review and suggested additions, deletions, and revisions.

KNOWLEDGE REQUIREMENTS PRE/POST DRIVING

Adjustments

Seat Position Mirrors

Occupant Protection

Restraint use Locked doors

Inspection

Signals Lights Tires Loose objects Behind vehicle

Cleaning

Windshields Lights

Securing Vehicle

VEHICLE CONTROL

Starting Starting procedure Limited warm-up

Accelerating

On the flat On upgrades On slippery surfaces

Upshifting (Manual

Transmission) Shift at proper Speed/rpm Coordinating Clutch/acceleration

Lane Keeping

Grasping wheel Adjusting wheel to turn to speed and position Fixate well ahead

Turning

Positioning for turn Adjusting speed for turn Turning wheel in relation to speed and path Straightening wheel

Regulating Speed Regulating accelerator to maintain speed Observing speedometer Keeping transmission in gear

Slowing/stopping

Anticipating stops Applying brake Easing brake at stop speed Maintaining brake pressure when stopped

Backing

Assuming proper body position Observing through rear window Coordinating clutch and accelerator Turning wheel in relation to speed and path Braking to a stop

RULES OF THE ROAD

Traffic Controls Traffic lights Stop signs Yield signs No-turn signs No enter signs Crosswalks Railroad crossing signs/lights Human controls enforcement/highway personnel)

Lane Control

Basic lane use Passing Reversible lanes Reserved lanes (eg, HOV) Shared left-turn lanes Backing Stopping One-way Lane drops, merges

Turns General rules Turn control signs Traffic Circles

Right-of-Way

Yielding right-of-way Intersections Traffic circles Pedestrians Emergency vehicles School buses

Vehicle restrictions

Parking Restrictions

VISUAL SEARCH

Maintaining Attention Maintaining general surveillance Avoiding distraction

Search Ahead

Distance Side-to-side

To the Side

Intersections Crosswalks Railroad crossings Roadside activity Sight obstructions Merges/on-ramps

Over-the-Shoulder

Lane change Merging

Mirrors

Periodic scanning When slowing Changing lanes Merging Overtaken on downgrades

Headlight Use

Use of high beams Dimming for vehicles Low beams for fog and rain Not retaliating

COMMUNICATION

Signaling Intentions

Signaling turns Nature Timing

Canceling signal Signaling slow/stop Uses hand signals when appropriate

Communicating Presence

Headlights Horn Emergency flashers Signals (reflectors, flares)

ADJUSTING SPEED

Compliance with Limits

Adjusting to Traction Slick surfaces Curves Hydroplaning

Adjusting to Visibility

Intersections Hills, curves Vehicles Weather Darkness Fog

Adjusting to Traffic

Prevailing speed Entering traffic Leaving traffic Pulls over when required Emergency vehicles

Specific Hazards

Maneuver limitations Roadside activity Path threats Pedestrian traffic Shopping areas Wildlife

POSITIONING VEHICLE

When Following

Vehicles in general Specific vehicles Limited visibility Avoiding blind spot Slippery surfaces When carrying/towing

heavy loads When followed

Passing Vehicles

Gap acceptance (2-3 lane) Lateral separation

Crossing/entering

Accepting proper gap Assuring clearance ahead Responding to turn signals Vision obstructed

When stopping/parking

Selecting locations Vehicle orientation Keeping clearance Observes restrictions

HANDLING EMERGENCIES

Vehicle Failures Brake Tire Headlight

Collision Avoidance Quick stop

Manual and ABS Quick turns Skid recovery Escape paths

Accident procedures

Scene control First aid Summoning help

DRIVER PREPARATION

Physical Fitness

Vision checks Hearing checks General physical checks Treatment for illness/disability Eating

General During trips

Exercise Fatigue prevention

Use of Alcohol and Other Drugs

Limiting consumption Limit of driving

Avoiding mixing Trip Planning Alternatives to Driving VEHICLE READINESS

Characteristics Vehicle size Engine size

Drive train configuration Displays (legibility) Controls (ease of reach, operation) Seats Trailers and towing

Safety Equipment

Passive restraints Mirrors Anti-lock brakes CB radio

Inspection/Maintenance Servicing

DRIVER MANUAL Licensing authorities in all states and provinces need to provide applicants with a written source of the information required to meet the knowledge requirements specified in the preceding section. The driver manual defines what is that drivers are expected to know and will be held responsible for knowing. These guidelines will address driver manual content, organization, and format.

Content — The subject matter of the driver manual should encompass, at the minimum, all of the knowledge requirements specified in the table above. In addition to content intended to benefit the safety and mobility of road users, other information may be included (e.g., licensing, driver services, organ donor program). Because the driver manual is so widely distributed to and read by the public, it serves as a valuable avenue of communication with the public. However, the inclusion of other topics should not compromise the ability of the manual to serve its primary function.

Organization — The driver manual is intended primarily to serve as a reference aid and should be organized in a manner that will facilitate access to the individual items of information on an as-needed basis. Such facilitation is achieved by use of: (1) relatively brief, self-contained sections, (2) headings that clearly identify the content of each section, and (3) a detailed subject index.

Format — The format of the driver manual should be designed to foster acquisition and retention of information. For doing so, it should make liberal use of:

• Short paragraphs • Bulleted phrases • Paragraph headings • Highlighting of key words

Reading Level — the information underlying proper vehicle operation is generally simple enough to be communicated at relatively low reading levels. The fifth- to sixth-grade reading level is considered optimum in communicating with the population of literate driver license applicants. The reading level of the entire manual should be checked before it is published. Available automated procedures for measuring reading levels may be applied quickly and inexpensively to computerized text.

Visuals — Pictures, diagrams, and other graphic displays frequently communicate better and give a more lasting impression than text. However, they tend to be expensive of space and should only be employed where they provide a clear benefit. Their use for clearly ornamental purposes is wasteful and can be deleterious wherever a limitation on the number of pages would force deletion of important content.

Practice Questions — the provision of practice test questions will enable applicants to assess their own knowledge and help those who are unfamiliar with the multiple-choice format to gain some experience in its use. It is best that practice questions not be drawn from the specific items making up the license test. To pass the test, applicants must study the whole manual, not just the practice test question.

KNOWLEDGE TESTS The primary purpose of a knowledge test in driver licensing is to assure the applicant's possession of the information needed to drive safely. It fulfills this purpose by (1) providing an incentive to applicants to secure from the driver manual the information needed to pass the test, and (2) furnishing a means by which applicants can demonstrate their possession of information. Test Construction The construction of knowledge tests will be discussed in terms of (1) content, format and wording of items, (2) scoring standards, and (3) alternate forms. Content of Knowledge Tests The knowledge test can only measure a sample of what applicants know. However, if the sample of items is sufficiently large, and represents the full-range knowledge requirements, the test will provide a reliable estimate of what applicants know about the subjects that make up the manual. The content of test items should be drawn directly from the driver license manual. If the manual has been designed to fulfill the knowledge requirements that underlie safe driving, so too will the test. Selecting test content directly from the manual also guarantees that the applicant will have had an opportunity to master it. To the fullest extent possible, the test items should be drawn from across all sections of the manual so that applicants will have reason to believe any item of information found in the manual may appear in the test. Wording items in the same manner as the text will help examiners demonstrate to applicants failing the item that it did indeed come from the manual. Item Format The multiple-choice-type of item offers the only practical means for testing large numbers of license applicants uniformly and objectively within the resources generally available to licensing agencies. The following considerations should guide the design of item format.

Content of alternatives — all of the alternative responses to an individual item should address the same piece of information and attempt to assess whether the applicants possess that information. If the various choices address different topics, there is no way of determining from responses what it is that an applicant does and does not know.

Correct and incorrect answers — each item should have only once correct answer; the rest should be clearly incorrect. Applicants should not be required to judge degrees of correctness (which is the "most correct" answer). Number of alternatives — Generally speaking, the greater the number of alternative responses, the smaller the chance of guessing the correct answer. However, the situation applies only where all alternatives are plausible. In driver license exams, it may be difficult to develop more than three alternatives that are plausible. Adding a fourth alternative that nobody chooses makes a test longer without making it better.

True-False — the true-false format should be avoided owing to (1) the relatively high probability of guessing the correct answer, and (2) differences in the interpretation of "true" and "false". Knowledgeable applicants are often scored incorrect because they know of exceptions to what are scored on the test as true statements.

Position of correct answer — the position of the correct answer in the series of alternatives should be decided by chance in order to prevent applicants from benefiting from systematic patterns, such as a tendency to put the correct answer in the middle of the series.

Sequence of alternatives — where alternatives follow a numerical or otherwise logical sequence, they should appear in that sequence on the test. To preserve the sequence, the alternative responses may have to be selected after the position of the correct alternative has been decided by chance.

Wording of Items Items should be worded such as to maximize the likelihood that applicants who know the answer will answer the item correctly and those who do not know it will answer incorrectly. In order to achieve this objective, the following should be avoided:

Complex words or phrases — the test should measure driving knowledge, not verbal skill.

"All of the Above" — in this type of question, all of the alternatives are actually correct. Applicants may read no further than the first alternative.

"None of the Above" — in those cases where this is the correct response, there is no way to determine whether an applicant knows what the correct answer truly is.

Legalese — what is written by and for lawyers is not necessarily understood by the public. Avoid legal terms and direct excerpts from the motor vehicle code. If the wording is taken directly from the manual, this will not be a problem.

Use of the negative form — A question that starts "Which of the following is not..." requires applicants to search for an incorrect answer. Knowledgeable applicants frequently forget this and choose what a correct answer is.

Inconsistent alternatives — Inconsistencies that attract attention to a particular alternative should be avoided, examples being alternatives that are substantially longer than others, use of attractive words such as "safely," or including a rationale for incorrect answers to make them appear more plausible.

Licensing authorities should make every effort to prevent applicants from passing the test simply by memorizing the answers to a limited number of test questions. The best means of achieving this objective is by drawing from such a large pool of test items that anything appearing in the driver manual may show up on the test. The availability of a large test item pool permits development of many alternative forms and, with computer testing, generation of a virtually unique test for each applicant. These practices prevent applicants from gaining high scores simply because they have taken the test before. Scoring Standards Individual States will decide the number or proportion of test items that must be answered correctly in order for the applicant to pass the test. Scoring standards should have relatively little effect upon licensing since almost all applicants will eventually and ultimately pass. What they can influence is the knowledge levels of the licensed population; the higher the standard, the more people will know. High Standards -- Fear of burdensome re-testing has encouraged some administrators to accept relatively low scoring standards; the idea that 75% is passing has a long history in education. A number of considerations encourage higher standards, such as 90%, for driver license test.

• Most of those taking the knowledge test are new drivers, lacking the skill and road savvy that comes with experience. For such a population, high standards can be justified.

• The content of the test is bounded by the content of a manual that can be mastered in a

few hours of preparation. Being apprised of the scoring standards in advance will encourage the preparation needed to pass the first time.

• A test is not completely accurate in measuring what applicants know. On a test with a

90% scoring standard, many applicants will pass the test knowing less than 90% of the information from which test items were drawn.

Differential standards — Recognizing that certain items of information are more critical to proper operation than others, agencies may wish to set differential standards, requiring close to a 100% for a subset of items in which lack of information poses a clear threat to the public safety e.g., traffic lights or right-of-way laws, and a 90% standard for most of the items.

Improving applicant performance — where an unacceptably high proportion of applicants fail to meet established passing standards, efforts should be undertaken to seek improvements in applicant knowledge and/or test procedure rather than lowering the test standards. Such improvements may include the following steps:

• Examining individual items to identify the specific items that are causing trouble,

• Revising the test to clarify any ambiguous questions and eliminate unnecessarily fine

distinctions, and

• Where the test items appear valid, revising the treatment of the corresponding subject matter in the driver manual, giving it greater visibility and/or improving the effectiveness of communication.

Alternate Forms Alternate forms of the knowledge test should be available for administration as retests, thus minimizing the chance of an applicant's being able to answer questions correctly because of previous exposure to the same questions on an earlier test administration. Development of alternate forms should adhere to the following:

Representative sampling — Each alternate form should sample representatively across all knowledge categories, in order to provide the best possible estimate of an applicant's total knowledge.

Independence of forms — the questions making up the various alternate forms should assess different items of information, not simply the same information with different wording. Getting a second chance to answer the same questions raises the score spuriously. If it is absolutely necessary to test for the same information on more than one form, at least the foils (incorrect alternatives) should be different.

Equality of forms — all test forms should have the same level of difficulty, as indicated by the mean proportion of items answered correctly. Equality may be achieved either by allocating individual items to forms on the basis of their difficulty level or through the use of norms that render the forms statistically equal.

Item independence — the various items appearing on any one form must be independent of one another. No question should be capable of being answered correctly solely on the basis of information supplied by another item appearing on the same form.

Re-testing — Since re-administration of the same test form may yield a spuriously high estimate of applicant knowledge, the chances of such occurring should be minimized by (1) keeping a record of the first to assure use of an alternate form on retest, or (2) having such a large number of alternative forms available that the chances of getting the same form twice are extremely small.

Computer generated tests — one means of meeting the requirements just described is by having the entire item pool stored in computers and individual items selected at random for each applicant. While automated testing obviously lends itself to such an approach, computers can print out copies for paper pencil testing. The possible number of test "forms" would be extremely large. With a large enough item pool and adequate test length; individual forms would be representative and equal in overall difficulty. The selection of items can be programmed to assure these conditions prevail and to prevent the answers to one item appearing in the stem of another question.

Item Analysis As a part of test development, item analyses should be carried out on each form of the test as a means of identifying deficient items. Each form should be administered to a representative sample of no less than 100 applicants prior to and following reading the driver manual. Items should be analyzed for response frequencies and item-test relationships. Response frequencies — the proportion of applicants answering each item correctly should be examined as clues to possible deficiencies in the wording of items. • Items with extremely low post-test pass rates and those showing little pre-post improvement

should be examined to make sure they are not misleading in some way. On the other had if the percent choosing the passing is close to 100%, wording should be examined to make sure that the correct answer is not being given away.

• The proportion of applicants picking each distracter (incorrect alternative) should also be

examined. Incorrect alternatives that are never chosen should be examined closely to see if they lack plausibility.

• It is common psychometric practice to seek items with close to a 50% pass rate in order to

maximize test variance and potential correlation with other variables. However, while controlling item difficulty is an acceptable practice in where the variable being measured is a hypothetical construct defined by the test itself, license test items are drawn from a defined body of content and pass-fail rates are what they turn out to be.

Item-test relationships — the relationship between performance on each item and total test score should be examined through the use of item-test statistics. Since all of the questions on the test came from the same source — the driver manual — applicants who do well on the test in general should also do well on each individual item. A weak item-test relationship suggests that something in the wording may be causing knowledgeable applicants to reject what is supposed to be the correct answer, or pick an alternative that also happens to be correct. The most common measure of item-test relationship is the item test correlation. Two shortcomings of a simple correlation coefficient in this application are (1) the difficulty in determining just how much of a relationship is due to the spurious effect of the item on total score and (2) the inability to detect which alternative is causing the trouble. A more informative technique is to compute the mean total score for the applicants selecting each of the alternative responses, subtracting "1.0" from the mean for the correct alternative to remove the effect of that item on the total score. A distracter having a higher mean score than the correct answer is a potential trouble source. Use of item analysis -- Where item analyses identify deficient items, every effort should be made to discover and remedy the deficiency. Wording of the items should be examined and individual alternatives rewritten identified. Generally, where the mean score of applicants picking a particular incorrect answer is higher than the score for those picking the correct answer, that alternative is likely to be the source of the problem. More often than not, something in the wording of that item is misleading knowledgeable examinees. Corresponding sections of the driver manual should also be scrutinized for messages that are might be unclear. However, no item should be deleted from a criterion-referenced measure such as a license test purely on the basis of item analysis results; the content of the knowledge test should be based upon what drivers need to know, not item statistics. Not every piece of information lends itself to the multiple choice format and some items will prove unsalvageable. However, exclusion should not be based upon the results of item analysis alone. Test Administration The way in which knowledge tests are administered can strongly influence their effectiveness in assuring the safety of drivers. Feedback to Applicants Time permitting, applicants can be apprised of their errors, informed of the correct answer and told where the information may be found in the driver manual. Where applicants have failed a test, this form of feedback may enhance the credibility of the test and forestall protests. While providing correct answers also serves a learning function, the benefits are relatively small since the questions with which applicants are confronted on any one test constitute a small portion of the total information for which they are held responsible. While applicants should be apprised of their errors, they should be advised that these errors are indicative of wider

information deficiencies that can only be remedied by studying the entire driver manual. Scheduling Re-tests Applicants who clearly fail the knowledge test should be required to wait at least a day before being re-tested. Some applicants may seek an immediate retest in the mistaken belief that they can pass simply by looking up answers to the questions they missed, an unlikely event where alternate forms are administered. Requiring a day's wait provides applicants an opportunity to restudy the manual in its entirety and thus prepare for any test form they might receive. Test Security Licensing authorities should make every effort to prevent copies of questions from falling in the hands of license applicants outside of the test situation. If applicants are permitted to take completed tests with them, the practice in some jurisdictions, all forms of the test will soon be in the hands of the applicant population, allowing the knowledge test to be passed simply by memorizing the answers to each test. If the tests addressed every item of information applicants were expected to know, such a practice would be acceptable. However, this is not the case since available questions rarely cover every item of information in a manual. The use of automated testing greatly enhances test security. Evaluating Knowledge Tests Any knowledge test must be evaluated against the purpose it serves. The purpose of a driving knowledge test in licensing is to foster safe operation of automobiles by assuring that drivers possess the knowledge needed to drive safely. It attempts to accomplish this by creating an incentive to secure the requisite information from the driver manual or other sources, and assessing the extent to which the information has been learned. Evaluation of a driving knowledge test includes measures of reliability, validity, and effectiveness. Stated briefly, reliability of a knowledge test is the extent to which each administration of the test estimates a person’s possession of the body of knowledge that defines the content of the test, validity is the extent to which the content of the test truly measures knowledge of safe driving, and effectiveness is the extent to which the testing process achieves the objective for which it takes place. As noted in the introduction, inappropriate application of concepts and techniques borrowed from psychological measurement has hampered the evaluation of driver license tests. Differences between psychometric and license tests in the means by which the reliability, validity and effectiveness of are evaluated will be highlighted in the following discussion.

Test Reliability The reliability of a knowledge test used in driver licensing is the extent to which it accurately estimate the body of knowledge that defines the scope of the test. The questions that make up a license test constitute but a small sample of what it is that drivers are expected to know. If the sample does not give a reliable estimate of an applicant's overall knowledge, the test may fail many applicants who really know enough to pass while passing many applicants who do not. The fact that a driving knowledge test has high reliability doesn’t mean that the knowledge being measured has anything to do with safe driving; that is validity. However, if the sample of knowledge making up the test doesn’t provide a reliable measure what a drivers know, it cannot possibly measure knowledge of safe driving. Estimating test reliability The reliability of knowledge tests, like that of any measurement process, can be estimated by comparing the results obtained from different samples of what is being measured. If the various items making up a test are accurately sampling what a person knows, then scores on individual items and groups of items should give similar results. The larger the sample of questions, the more similar should be the results and the more reliable the estimate of knowledge. If different samples give widely differing results then the score individual applicants receive will depend more upon which questions they were asked than how much they know. Use of alternate forms - The reliability of the entire test can be assessed where alternate forms of the test are available. If each form provides an accurate measure of total knowledge, then scores on alternate forms should correlate highly with one another. Reliability can be estimated by having a group of applicants take all forms of the test and comparing test results across the different forms. Split-half measures -- Where alternate forms are not available, reliability may be estimated by comparing scores obtained from items making up one half of the test with scores from the other half and statistically projecting the reliability of the entire test . The most commonly used procedure for doing this is the Spearman-Brown Prophecy formula. In estimating “split-half” reliability is best to compare odd number with even numbered items, rather than the first and second halves of the test, in order to equalize the content of the two halves should the items follow any ordered sequence. Item-test relationships -- Kuder-Richardson formula 20 estimates the internal consistency of a test by adding individual item variances (the proportion passing times the proportion failing an item) and subtracting the sum from total test variance to furnish the covariance among items. The KR-20 formula is easily applied with available computer programs and saves both the labor and possible bias involved in splitting a test in half.

Expressing reliability --The most common way of expressing reliability is the product-moment correlation between forms or halves of a test. However, limitation of correlation is its sensitivity to test variance; the lower the variance of any two measures being correlated the lower will be the correlation. Once a license test is in use, if the manual is successful in ensuring all applicants possess all or almost all of its content, variance of test scores could become restricted to the point that correlations among test forms are quite low even though the tests might be estimating knowledge very reliably. The interpretation of correlation must take test variance into account. An alternative expression of reliability is the standard error of measurement, which provides an estimate of how accurately scores on the test estimates total knowledge. On a test with a standard error of measurement of ±5%, an applicant’s score will come with 5% of what the applicants knows about two-thirds of the time. The advantage of using standard error of measurement is that it is relatively unaffected by test variance and its meaning is easily understood. Test equivalence -- Various forms of tests may correlate highly with one another, such that people who score the highest on one form also do so on others, and yet vary in difficulty of the items so as to yield differences in average scores, or in the range of scores. While this type of error can be overcome by converting raw scores to percentiles or standard scores, it is more conveniently avoided by assigning items to forms in a way that will make the different forms approximately equal in difficulty and variance to begin with. Test length and reliability -- The most direct way of achieving acceptable reliability is by assuring that the numbers of items making up each form of the test are large enough to provide reliable samples of what applicants know. A direct way of finding out how large a test must be is by giving the full set of questions available to a group of drivers and seeing what happens to the correlation and standard error of measurement when the items are divided among more and therefore shorter test forms. Where the correlations begin to drop off and the standard error begins to rise provides a clue to minimum acceptable test length. While there’s no fixed number of items for a reliable knowledge test, somewhere in the neighborhood of 40 - 50 items are usually needed for a reliable result. Test Validity The validity of any test is a function of the extent to which it measures what it purports to measure. There are a number of means by which the validity of tests is assessed. Content validity The form of validity most appropriate content validity that is the extent to which the test truly assesses an applicant's knowledge with respect to a defined body of informational content.

Source of content -- In most States, the documented record of what drivers are expected to know is the driver manual. However, the content of the test as a measure of what it takes to driver safely depends upon the source of the content itself. The knowledge requirements specified earlier were derived from a systematic analysis of driving tasks and an evaluation of their criticality to driving safety. A test based upon these requirements would be content valid to the extent that the task analytic process yielded an accurate picture of what drivers must do, and inferences as to the knowledge underlying them were accurate. Until a more systematic process produces a different outcome, the test offers as much content validity as is currently possible. Public acceptance -- To the public taking a knowledge test for licensing, content validity is defined practically by the content of the manual they used in order to prepare for the test. This definition becomes apparent when examiners are called upon the justify questions to applicants who answer them incorrectly. When asked "show me where it says that!" the ability to point to page and paragraph becomes as practical measure of validity. Content validity can be enhanced by seeing to it that items are drawn from the full range of information defining manual content and are expressed in largely the same terms as are used in the manual. Predictive validity If knowledge plays a part in preventing accidents, then the most knowledgeable drivers should have the fewest accidents. The validity of driving knowledge tests has often been assessed by correlating test scores with indices of subsequent driving to see how well the scores predict who will have accidents, violations or others indices of unsafe driving. However, predictive validity is less than appropriate to evaluation of driver knowledge tests in a licensing application for three reasons.

Multiple causation -- A serious obstacle to the use of correlation in evaluation measures is that the knowledge measured is one among a large number of influences upon driving behavior and therefore upon accidents. Many variables have a much stronger influence upon driving than knowledge and can easily conceal its effect. For example, if drivers with high scores happened to drive more than those with low scores, they could show larger numbers of accidents. Some of the influences such as annual mileage, age, gender or education, can be controlled statistically, at least to some extent. However, many of these confounding influences cannot be measured or even estimated, with the result that effects of knowledge can be difficult if not impossible to isolate.

Restriction in variance -- The fact that applicants must pass the license test before being allowed to drive means that the variance of scores achieved by licensed drivers is necessarily limited. If a driving knowledge test were totally effective in leading to the acquisition of knowledge everyone would ultimately score 100% on it, test variance would be zero as would be the correlation of scores with accidents -- or anything else. In practice there is little chance that all license applicants will answer all items correctly on any test. However,

the restriction in range of scores for those passing the test compounds the problem of multiple causation, further complicating the assessment of tests.

Purpose of tests -- As noted in the Introduction, the primary purpose of a driver knowledge test in licensing is not to predict future driving but to improve it by inducing license applicants to acquire the requisite knowledge and to assure that they possess it before they are allowed to drive. It is a quality control measure, functioning in the same manner as a final exam in a school course. The correlation between test score and behavior does not necessarily reflect its ability to change behavior. Evaluating the effectiveness of a knowledge test in improving driving safety is an experimental rather than correlational process and will be addressed in the next section.

Using prior driving record as a validation criterion avoids the restriction in variance that results from the use of test results for licensing purposes, but not the lack of control over confounding influences. Nor could prior driving records be used to validate tests for novice drivers, who have no such records. Effectiveness of Knowledge Testing As noted in the previous section, one way in which a knowledge test is expected to contribute to driving safety is not so much by predicting who will be unsafe drivers but by creating an incentive for them to acquire the knowledge needed to drive safely. The test, manual, and other sources of information form a process that can be evaluated for its effectiveness in helping to assure the safety of new drivers. To assess effectiveness in absolute terms would require having a group of drivers be licensed without having to take the test, and comparing their safety with that of drivers required to take the test. No State has yet been willing to allow novice drivers to operate vehicles without some test of their knowledge in order to conduct such an assessment. The effectiveness of improvements in a manual and test can be assessed by assigning the old and new items at random to large samples of drivers and comparing their subsequent accident and violation records. Several studies have found improvements in license manuals and tests for operators of automobiles and motorcycles associated with lower accident rates. Given the relatively small influence that pure knowledge plays in accident causation, and the many sources of information available to drivers, the effects of improved manuals and tests is expected to be small. The small expense of the licensing process relative to the costs of accidents makes even small improvements in safety highly cost-beneficial. However, the degree of experimental control and the large samples of drivers needed to detect small effects complicate evaluation of effectiveness. The complications of distributing different manuals and test to individual applicants on a random basis, making sure each applicant is tested on the appropriate materials are very great, making control at the level of individual applicants difficult to achieve, although it has been done in several evaluations of license manuals and tests. An alternative is to assign branch offices at

random to alternative manuals and tests, following up the subsequent accident and violation records of drivers tested at each location. Control over differences among locations could be achieved by comparing change in accident/violation rates for locations in which new manuals and tests were introduced with change (or lack of it) for locations retaining the original manual. Experimental evaluation involving control groups involve ethical and legal concerns as well those of a scientific nature. The fact that the control group is deprived of a process that might reduce its risk of death, injury, or property damage could expose those conducting the evaluation to liability for such consequences. An acceptable defense is that the benefit of the process is unknown prior to the experiment and those in the control group are not being denied anything of proven value. Indeed, the experimental process might prove to be detrimental. The only time that agencies have been considered liable for the negative consequences of control group assignment is when the experiment has been allowed to continue beyond the point that such consequences were known. However, further protection is offered by asking drivers for consent to become part of an experiment, allowing those with objections to opt out. Only those giving informed consent would become part of the experiment and randomly assigned to new versus existing manuals and tests. Automated Testing Electrical and electronic devices have been developed and widely used to automate the administration of knowledge tests. While a wide range of automated devices is available, almost all display test questions on a screen and require responses to be registered through some mechanical device. Two basic methods of displaying questions are through computer-generated images and through video images stored on computer-controlled CD-ROM, videodisc or videocassette players. Automation offers a number of potential advantages over written testing: Processing — an advantage of automated testing is reducing the labor associated with scoring tests. Since scoring of written tests generally requires but seconds, the labor saved is limited. However, the same automation also permits results of testing to be entered into drive records and stored for statistical purposes. There also exits the possibility of using the same computer-controlled displays to test visual acuity and other visual functions, and even allowing some applicants to enter identifying and personal history information directly into the driver license file. Feedback — Automation allows applicants to be given correct answers after they have responded and their answers recorded. While informational feedback contributes to the acquisition of knowledge, the size of the contribution is minimal. Since the items of a knowledge test comprise but a small sample of what drivers are supposed to know, filling the specific information gaps revealed by any one test administration is but a small step toward overcoming the full range of knowledge deficiencies. Probably the greatest value of feedback is relieving examiners of the need to explain and justify answers to applicants who fail the test and ask to review their results,

an activity that can be more demanding of examiner time than scoring the tests. Individualization — Automation allows different sets of test items for individual applicants. The advantage of such individualization is two-fold. First, it prevents applicants from knowing in advance the specific set of items on which they will be tested, thereby rendering the test largely cheat-proof. Second, it allows the use of "adaptive" sequencing in which the order of items is determined by applicant performance, applicants need not complete the entire test if the responses to the first of questions are such that their probability in passing the entire test is close to 100%. The time saved frees up the test equipment for other applicants, thereby making a more efficient use of the equipment. The test could also be terminated as soon as they get enough incorrect answers fail, although public relations considerations may dictate allowing them to complete the entire test. Imagery — the electronics of most automated equipment allows detailed static and dynamic images to be displayed in full color relatively inexpensively. This capability is particularly valuable in presenting information concerned with driving. Questions about driving situations can be presented in the same way they occur on the road, allowing complex situations to be addressed without placing demands on verbal skill. If applicants know what to do in actual driving they will be able to answer questions correctly, a requirement for valid testing that often cannot be achieved through written knowledge tests. SPECIAL APPLICANT CATEGORIES The requirements that have been described to this point apply to Basic Driver's License applicants in general. Additional requirements are created by the needs of reading-limited and foreign-speaking applicants. Reading-Limited Applicants License applicants may possess the fundamental abilities needed to operate a motor vehicle safely and yet be unable to pass a driver knowledge test due to (1) inability to read a driver manual and acquire the information needed to pass the test, and/or (2) inability to understand written test questions well enough to pass a driver knowledge test even when they possess the requisite knowledge. This section will address the special needs of applicants whose language limitations are such that their needs cannot be met through the driver manual or driver knowledge test. Providing Information to the Reading-Limited Applicants who are unable to read at a fifth or sixth-grade reading level do not have available to them a source of information needed to meet knowledge requirements such as is provided to literate applicants. While some reading-limited applicants are able to meet knowledge requirements with the aid of friends or instructors, others of them either drive unlicensed or manage to pass tests and obtain licenses without really meeting knowledge requirements.

Providing reading-limited applicants some way of acquiring information not only eases the burden upon applicants but helps to assure the safety and mobility of the public. Two alternative methods of providing information that have been used effectively with reading-limited applicants are audio and video presentations.

Audio presentations — The information needed to fulfill knowledge requirements can be recorded on audio cassettes that can be loaned, rented, or sold to applicants for aural delivery of information along with printed materials to present pictorial and other content that cannot be adequately communicated through the spoken word. The audio and print material must be designed specifically to meet the needs of reading-limited applicants. Simply reading the text of the driver manual into a cassette will not suffice. Effective learning and retention will require modification of language and sentence structure, integration of the audio and visual presentations, and provision for review and self-testing. The assistance of specialists in teaching the reading-limited should be engaged in preparing cassettes.

Video presentations — with the widespread availability of videocassette recorders in homes and libraries, video offers a practical and effective means of communicating to the reading-limited. The increased availability of personal computers opens up the future use of CD-ROM and Internet, media that not only handle visual information but allow interactive forms of instruction to accommodate individual differences in learning and provide for self-testing to help assure learning. However this medium is not well suited to use of sound at the present time. Because the inability to read is often symptomatic of difficulty in handling verbal symbols at all, whether written or spoken, the ability of video to communicate driving-related information in graphic and dynamic form enhances its ability to communicate. Video cassettes and CD-ROM disks may be loaned or rented to applicants, and made available through schools and rental outlets. The availability of a video version of the driver manual has been shown to yield large and significant increases in knowledge levels of reading-limited applicants.

Testing Reading-Limited Applicants License applicants whose reading limitations place them under a handicap when taking a written test should be provided an alternative mode of testing that will yield an accurate estimate of their knowledge despite their limitations. Such alternatives include oral tests, pictorial tests, and audio-visual tests. Oral Tests — Reading-limited applicants may have the written test read to them by an examiner, who also registers their selection of answers. However, there are several drawbacks to oral testing: (1) the spoken word is incapable of handling questions dealing with highly visual content, such as signs and signals, (2) even where reading-limited applicants are able to understand questions, an oral test still places them under a handicap in that they are unable to weight the alternative responses simultaneously in selecting among them, (3) the oral testing process can be influenced by the examiners, who are known to provide aid that is unavailable to an applicant taking a written test, and (4) oral testing is extremely labor-intensive and therefore

an order of magnitude more costly than administration of written tests. For these reasons, simple oral testing is not recommended as a means of examining reading-limited applicants. Where it is employed, the pass-fail rates of individual examiners should be reviewed periodically to identify those whose averages deviate sharply from others in the same office. Pictorial Test — the handicap imposed by the inability of oral examiners to view and review alternative responses simultaneously can be overcome through pictorial representations of each alternative. While each alternative must still be read to the applicant, the pictorials serve as stand-ins for test text. Applicants can consider the alternatives at length and point to the correct answer in order to register their selection. As with written tests, alternative forms of the pictorial tests must be available in order to keep applicants from passing tests by memorizing answers rather than learning the material. While it overcomes one limitation of oral testing, use of the pictorial test is still very labor-intensive and highly subjective. If licensing agencies are willing to abide the cost, the pictorial oral test is acceptable so long as examiners administer it objectively. The pass-fail rates of individual examiners should still be reviewed periodically Audiovisual Test — the addition of sound to automated testing provides a means of testing reading-limited applicants. Present day CD-ROM, videodisc and videocassette technology allow questions to be presented objectively and understandably in audiovisual form. Moreover, the ability of the visual presentation to display motion helps overcome any limitations of applicants in their ability to infer motion from the spoken word or from still pictures. Because the system is entirely automated, it (1) renders oral tests as objective as written tests, and (2) virtually eliminates the personnel costs of oral testing. An effective audiovisual test possesses the following characteristics:

Presentation of questions — The question and alternative answers can be displayed serially on a video display terminal, after which all alternatives can be displayed simultaneously in split-screen form. As the narrative for each alternative response is repeated, the corresponding frame can be highlighted in order to assist applicants in associating answers with corresponding frames. Where the visuals involve motion, the split-screen presentation should select key frames that clearly distinguish one alternative answer from another.

Registering answers — Applicants may register their answer either through a special answer pad or by touching a touch-sensitive screen. The latter, although the more expensive approach, is easier for applicants to understand and minimizes errors in registering responses.

Question review — Applicants must be given an opportunity to review the question and alternative answers, just as a literate applicant can do on a written test. This can be handled by providing the applicant a means of repeating the question as desired. The number of repetitions should be limited in order to prevent applicants from tying up the equipment for long periods of time while reviewing questions. Where applicants are unable to answer a question in a reasonable length of time, the problem is very likely to lie in ignorance of the correct answer rather than inability to understand the question.

Summoning help — Applicants must be provided a means of summoning help if necessary. However, such requests must be discouraged if the advantages of automation are to be fully realized. If applicants allow a long period to elapse, (e.g., 30 seconds) without either answering a question or requesting repetition, a message should tell them how to summon assistance. Examiners must determine whether the request for assistance results from difficulty with equipment or merely ignorance of the correct answer. They should anticipate that applicants who are not adequately prepared will tend to attribute their shortcomings to the equipment.

Foreign-Speaking Applicants The inability to read or speak the English language is not necessarily a barrier to proper motor vehicle operation so long as drivers meet prescribed knowledge requirements and are able to interpret highway signs, signals and markings. It is the responsibility of the licensing agency to assure that these conditions are met before issuing a license. Fulfilling this responsibility imposes special requirements. Given the importance of mobility to the welfare of the individual, the inability of the foreign-speaking to pass the regular knowledge test will often lead to their gaining license fraudulently or operating a vehicle without one. Therefore, steps taken to accommodate the needs of foreign-speaking applicants will help prevent unqualified drivers from threatening the safety and mobility of the motoring public. Presenting Information to the Foreign-Speaking A foreign-language version of the driver manual represents the most direct way of presenting information to the foreign-speaking.

Preparing materials — where the numbers of applicants speaking any one language are large, copy can be printed in volume, the same way as the English-speaking version. However, where the population is small, copy can be reproduced in limited numbers on an as-needed basis. A relatively inexpensive alternative to printed materials with small populations is an audio cassette translation to accompany the regular manual (for graphic material). Cassettes could be made available on a purchase or loan basis.

Obtaining Assistance — the cooperation and assistance of nationality- and language-affiliated groups often can be enlisted in preparing materials for their constituency. Such voluntary assistance is beneficial in (1) minimizing costs involved in translating information, (2) resolving issues involving dialect and usage for different countries and regions speaking the same language, and (3) enhancing the credibility of materials among users. Where licensing authorities are unable to assume the burden of preparing a foreign language manual, cooperating organizations may be encouraged to assume the responsibility of printing and distributing materials.

Testing Foreign-Speaking Applicants Foreign language versions of written test forms can be prepared with volunteer group assistance as was suggested for informational materials. Steps that should be taken to ensure the equitable testing of foreign-speaking applicants include the following:

Alternate forms — foreign language versions should be prepared for each of the alternate forms available in testing English-speaking applicants. Where only one form is available, foreign-speaking applicants frequently succeed in memorizing the order of answers.

Oral testing — where foreign language versions of tests are unavailable, or where applicants are not literate in their own language, oral testing may be the only way to assess knowledge. Applicants should not be permitted to bring their own interpreters, who may provide answers rather than questions. If no examiner speaks the language, arrangements should be made to employ, or obtain volunteer assistance from a reliable independent interpreter.

Audiovisual testing — if an audiovisual test is available for examining reading-limited applicants, the addition of foreign language sound tracks will provide a cost-effective alternative to oral testing. Volunteer narrators may be used to prepare soundtracks in the same manner as suggested for written test questions. Authenticity is critical; vocal qualities are not.

GUIDELINES FOR SKILL TESTING The purpose of a skill test, as addressed by these guidelines, is to assess an applicant's possession of the skills that are required to operate an automobile in a manner consistent with the safety and mobility of the motoring public. A skill test cannot predict how safely people actually drive when they are not being tested. The way people behave on the highway is determined to a great extent by attitudes and habits that are seldom revealed in the presence of a license examiner. Indeed, research has shown there is no relationship between the driving practices shown on a driving test and those that occur after an applicant leaves the license station. What a skill test can do is to force applicants to acquire requisite skills through instruction and practice, and to assure possession of these skills, before they are issued a license to operate unsupervised. Skill Requirements A driving "skill", as defined by these guidelines, is an ability that requires both knowledge and practice for its attainment1. The specific driving skills addressed by the guidelines are those that are considered critical for the safety of both the driver and other road users. The guidelines do not deal with skills needed solely for traffic flow, fuel efficiency, or passenger comfort. The skills that are required in safe automobile operation are divided into the following three categories:

Perceptual — the ability to perceive characteristics of the highway traffic environments in a way that permits safe vehicle operation, e.g. judging gaps, identifying hazards.

Attentional — the ability to focus and shift attention, e.g. to monitor traffic ahead and to the side in a merge.

Motor — The ability to manipulate controls in order to maneuver the vehicle, e.g. ability to rotate the steering wheel in relation to the motion of the vehicle and intended path when turning a corner.

While basic cognitive skills are required to learn and apply facts, procedures, and principles, driving-specific cognitive skill are primarily concerned with navigation rather than safety. Specific skills in each category are summarized on the next page. Advanced skills, such as those involved in rapid stops or controlling skids, are not among those currently included in driver license testing. Their benefit to safety is open to question, as is the

1 Driving specific skills are distinguished from basic sensory, attentional, perceptual, cognitive and

psychomotor abilities. While tests for the latter are appropriate to assessment of individuals who may be deficient with respect to certain abilities, they lie outside the scope of initial license testing. The term "skill" has also been used to mean the level of some ability, e.g. to have “a lot of skill.” To avoid confusion, the term “proficiency” will be used to refer to ability level.

ability of new drivers to master them. However, the introduction of graduated licensing systems, along with development of devices to permit safe and economical testing of the advanced skills described may change this situation at some time in the future. Attentional Skills

Attention-sharing — Controlling and maneuvering a vehicle while attending to traffic controls and other road users (search, signaling, and space management)

Attention shifting — Shifting attention as needed (ahead, to the side, and to mirrors)

Perceptual Skills

Spatial judgment — Judging the nature and magnitude of changes in speed and direction of other road users

Gap judgment — Judging the adequacy of gaps when merging, crossing, or entering traffic

Distance judgment — Judging the adequacy of distance of an oncoming vehicle when passing

Hazard detection — Detecting hazards in the characteristics and motion of other road users and in the roadway environment

Routine Motor Skills

Acceleration — Regulating pedal force to accelerate on level and inclined surfaces.

Shifting — Coordinating clutch, accelerator, and shift lever if manual transmission is used.

Maintaining speed — Regulating accelerator force in order to maintain a steady speed

Lane keeping — Coordinating speed and steering in order to keep the vehicle position within lane on straight and curved paths.

Turning — Coordinating speed and steering when turning corners.

Slowing — Regulating brake and accelerator to reduce speed

Stopping — Coordinating brake, accelerator (and clutch) to bring the vehicle to a stop at a given point

Backing — All of the above in moving the vehicle backward

Adjusting to limited traction — All of the above when operating on slippery surfaces

Advanced Motor Skills

Quick stop — Regulating braking to stop in the shortest possible distance

Swerving — Turning sharply to avoid a collision while maintaining vehicle control

Skid control — Maintaining directional control of the vehicle during a skid

Skid recovery — Regaining directional control following a skid

Evasive Action — Initiating an appropriate escape action when threatened with a collision

Testing Requirements Skills cannot be measured directly, but are inferred from performance in carrying out various deriving tasks. To yield reliable and valid measures of skills, the performances making up a skill test must meet the following requirements:

Uniformity — all applicants must receive essentially the same test, no matter when or where they take it. Applicants with the same ability should have the same probability of passing the test.

Objectivity — the scores received by applicants should reflect their performance to the

greatest extent possible and be influenced as little as possible by examiners. Two examiners observing the same performance should score the applicant in the same way. Scores should, as much as possible, be reckoned in terms of times, distances, and whether or not a particular response occurred, rather than a subjective judgment, with results totaled such that the applicant's score accurately indicates the level of performance.

Public Acceptance — any skill test must be acceptable to the public in that it must appear to be a valid measure of necessary driving skills, must avoid subjecting applicants to stress or unnecessary embarrassment, and must not expose them to danger.

Testing Modes These guidelines encompass three modes of skill testing:

Road testing — Observing and recording the performance of applicants operating their own vehicles over prescribed routes under prevailing traffic conditions.

Off-street testing — Observing and scoring the performance of applicants carrying out prescribed maneuvers under controlled conditions in an off-street area.

Simulation — observing the performance of applicants responding to highway traffic conditions simulated through displays and controls that demand of drivers the same skills required in operating an automobile.

The various testing modes are suited to assessment of different skills and require different procedures. Therefore each is addressed separately. ROAD TEST The most commonly employed method of assessing driver skill is through examiner observations performed while the applicant operates a vehicle on public streets and highways. The widespread use of road testing as a measure of driving skill reflects its several strong points. First, since it represents a slice of driving itself; the skills assessed in a road test are those required in everyday driving. Second, the applicant's familiarity with the vehicle in which the test takes place allows performance on the road test to provide a more accurate reflection of driving skill than would a test taking place in a simulator. Finally, the test is given in the applicant's vehicle on public streets, avoiding some of the costs incurred in simulator or closed course tests. Testing costs are limited to those associated with the examiner's time which, of course, is not insubstantial. . Applicant skill is inferred from performance in responding to highway traffic conditions encountered during the road test. The road test itself consists of applicant performances that examiners are to observe, the criteria that distinguish acceptable from unacceptable

performance, routes over which the test takes place, administrative procedures, and a method of scoring performance to determine whether applicants have passed or failed the test Purpose of a Road Test The purpose of a road test in driver licensing is to assure that drivers have sufficient skill to be allowed to operate a vehicle without supervision. It is not a test of driving practices or habits. Research has shown that there is no relationship between the extent to which drivers demonstrate such practices as signaling, checking the mirror or staying within the speed limit during a road test, and their use of these practices when they are not being tested. The only test performances that correlate with normal driving are those that require the development of skill, such as the maintaining the right path in turns and curves, or stopping at the stop line. Checking such non-skilled practices as signaling provide indirect measures of skill in that drivers who know they are supposed to signal, having passed the written test, and fail to do so on the road test tend to be those whose vehicle handling skill is so marginal that they are unable divide their attention between driving and observing various safe driving practices. In order for the road test to provide a measure of skill, drivers must be informed in advance as to the maneuvers that they will be called upon to make and the specific performances that will be scored. Having to take the test and fail it just to find out what it consists of wastes the time of applicant and examiner. Applicants should not, of course, be advised of specific test routes since it would allow them rehearse their performance until they can be performed by rote memory, a luxury they won’t enjoy in daily driving. Road Test Performances Almost all driving performances provide some measure of applicant skill. Performances that require the attention-sharing, perceptual, or motor skills described earlier provide direct measures of skill. As noted, other performances that do not require special driving skills for their execution do provide indirect measures of proficiency in certain skills by requiring applicants to perform them while applying their driving skills at the same time. Various examples are visual search, signaling, and obeying traffic signs or signals. Applicants who fail to do these things also tend to be those whose skills are marginal and therefore demand most of their attention. Selection of Performances for Observation The heart of the road test is the set of performances that examiners are to observe and evaluate. No examiner can observe and evaluate everything an applicant does. The range of performances capable of influencing the public's safety is too great, and the rate at which they occur during a road test too rapid, to expect examiners to score them all. To be objective and uniform, the road test must identify the performances an examiner is to observe as well as the locations where they are most likely to occur. This is the road testing approach that has been used in several research-based road tests developed for licensing purposes, including the Michigan Driver Performance

Measure, the USC Safe Performance Test, the Automobile Driver On-Road Performance Test (ADOPT) and the Commercial Driver License (CDL) Test, as well as the Motorcyclist In-traffic Test (MIT). It is also the approach employed by the California Driver Performance Evaluation (DPE), which forms the basis of the Model Road Test noted in the Preface to these guidelines. Where the examiner’s attention is not directed at specific performances, many of them are overlooked. Also since most examiners tend to notice errors more than successful performance, applicants may be scored primarily on what they did wrong, and their chances of failing the test increase with the number of situations they encounter. Tests intended primarily for self-evaluation or improvement are not subject to the same need for objectivity as are tests that determine eligibility for a license. In deciding what performances should be scored, the following need to be considered:

Opportunity — Situations requiring each performance must occur with sufficient regularity to assure that all applicants are scored on the same set of performances. Looking for performances that depend upon particular traffic conditions or weather conditions tend to be unproductive as well as detracting from the uniformity of the test.

Objectivity — Performances that can be assessed objectively are to be preferred over those that require subjective judgment on the part of the examiner. For example, signaling, checking a mirror, or staying in lane can be defined far more objectively than slowing for a “hazard.”

Safety/Mobility — Performances that cannot be tested without threatening the safety of the applicant, examiner, or other road users, or obstructing traffic, are more appropriately assessed off-street, if at all. For example, assessing the ability of applicants to swerve sharply is likely to threaten the safety of road users, while parallel parking may tend to obstruct traffic.

Road Test Maneuvers Even when the performances to be observed are very limited, no examiner can possibly observe them all. Whether or not examiners observe a particular performance depends greatly upon whether their attention is specifically focused upon it. One way of assuring that attention will be properly focused is by identifying the sequence of performances as they occur in specific maneuvers. Maneuvers that involve a set series of performances include: ⋅ R/L turns without cross traffic

⋅ left turn, oncoming traffic

⋅ merge

⋅ straight driving

⋅ R/L turns with cross traffic

⋅ negotiating a curve

⋅ exit

⋅ traffic responses

⋅ straight across traffic

⋅ lane change

By memorizing the performances associated with each maneuver, examiners will be assured of

directing their attention to the right place at the right time. Moreover, structuring an examiner score sheet according to the sequence in which performances occur during a maneuver facilitates recording applicant performance. Research shows that, by concentrating their attention on a specific set of performances at the point where those performances are most likely to be required, examiners will actually see more than they will if they try to observe everything. To achieve the greatest possible uniformity in testing, the maneuvers making up the test, and the number of times each maneuver is to be performed, should be the same for each test administration. Every applicant in every location throughout the State should, to the extent possible, face the same array of maneuvers. Such will not always be possible; there may be no freeway or other location for an angle merge or exit maneuver. Where necessary, other maneuvers will have to be substituted to at least require the same number maneuvers. For example, a lane change will require signal, search, speed and control performances similar to those required in a merge. Road Test Performances Within each of the maneuvers, a set of performances can be listed in the order by which they occur during the maneuver. These performances form the basic elements of the road test, and include the following: Signal -- for turns, lane changes, merges, exits

Search -- in turns, lane changes, merges, straight driving

Entry position -- in turns, curves

Speed -- in turns, curves, merge, exits, straight driving

Entry speed -- turns, curves

Path -- in turns, curves, merges, exits

Full stop -- stop signs, traffic

Lane selection -- in turns, straight driving

Stop position -- stop signs, traffic lights

Lane position -- straight driving

Gap judgment -- cross/enter traffic, lane change

Following distance -- straight driving

Traffic Dependent Performance Traffic conditions along the test route may require performances that cannot be anticipated (e.g., following a vehicle ahead, responding to a pedestrian crossing the street). Because there is no way of knowing in advance just where traffic-dependent situations will arise (1) the attention of examiners cannot be directed toward them to assure objectivity of scoring and, (2) the number and nature of situations will vary from one applicant to another, making it non-uniform. If the responses of applicants to such situations indicate that they are a clear danger to the public, provision can be made for an “immediate failure” (see below). Lesser dangers can be scored, without detracting greatly from test uniformity, by providing one catch-all check for all responses to unplanned traffic conditions.

Immediate Test Failures Examiners should be permitted to fail an applicant on the road test immediately in the event of performance demonstrating a skill deficiency that is sufficiently great that the continuation of the test is not only unnecessary but may place the applicant, examiner, or motoring public in jeopardy. Such performances would include, but not be limited to, running a red light, driving at extremely high or low speed, driving the wrong way on a one-way street or off-ramp, or requiring intervention of the examiner to prevent an accident. Some jurisdictions fail applicants immediately for violation of any law. However, since almost all road test performances are legally required, the strict adherence to this practice would result in failing applicants for any error, including failure to activate or cancel a turn signal. If the purpose of a road test is to measure skill, then failing on a technicality any applicant that possesses the required skills serves no purpose except to require the state to give, and the applicant to take, an additional unnecessary road test. Criteria for Assessing Performance Examiners must be provided a means to assess the adequacy of applicant performance. Scoring the road test must include, for each performance, criteria that will allow the examiner to distinguish acceptable from unacceptable performance. In educational settings, where test results are to be used only to guide instruction, a purely subjective appraisal may be acceptable. However, where test results determine issuance of a driver license, the criteria must be objective and uniform. Two applicants with the same skill should receive the same score, no matter who gives the test. When test criteria are subjective, examiners may score the same performances differently, with the result that the unqualified may pass the test and become a danger to the public, while qualified drivers may fail and have to be re-tested unnecessarily. The establishment of scoring criteria must take into consideration the variety of situations under which performance is observed, across different routes and at different times of day on the same route. The criteria must be broad enough to apply to virtually all conditions under which the test might be given. They must also be relatively simple; an extremely complex scoring system, or one that relies on examiner judgment, will rarely result in a uniform test. Take, for example, signaling a lane change. Simply requiring the signal be given before initiating the maneuver would be objective and simple but would not assure adequate warning to others. On the other hand, attempting to prescribe precisely when the signal should be given, would require taking account of so many road and traffic conditions as to become extremely complicated, or leaving the question entirely to the judgment of the examiner. While setting a fixed minimum time interval, such as 3 sec. may appear arbitrary, it is uniform and, when communicated to applicants in advance, entirely equitable. Route Selection

All road testing should take place over specified routes. Examiners should not be called upon to make up routes during a road test. Only by designating routes in advance is it possible to maintain uniformity in testing.

Number of routes — several routes should be devised for each license testing station. Having only a few routes allows applicants to practice driving each route to the point that their performance reflects their memory of the route rather than their general driving skills.

Route length — Fifteen minutes of driving in typical urban-suburban settings typically allows for approximately 150 observations of the driver performances making up the maneuvers that have been listed. This number of observations is a minimum for reliable estimation of a driver's skill.

Route conditions — Areas characterized by many traffic lights, heavy vehicular travel or pedestrian traffic should be avoided since they can introduce lengthy delays. The number of performances that can be assessed under these conditions is too few for the time spent. Where temporary conditions (e.g., construction/repair) interrupt traffic or change performance requirements, use of the route should be abandoned until normal conditions resume.

Maximizing observations — the various maneuvers differ greatly in the opportunities they provide to observe the performances that involve skill. For example, turning, particularly at uncontrolled intersections, permits much more opportunity to assess driver skill than does straight driving. Routes need to be chosen in a way that will result in maneuvers that maximize the opportunities to observe scored performance.

Setting up routes — a test route should be viewed as a path between maneuvers. Locations that permit the full array of maneuvers should be selected first and routes planned to interconnect these locations. The art comes in finding enough of the right locations without requiring frequent or long stretches of straight driving, which provide little opportunity to test skill.

Separating observations — Locations at which performances are observed need to be sufficiently far apart to allow examiners time to record applicant performance. For example, requiring two turns a block apart may not allow the examiner enough time between the last observation of the first turn and the first observation of the next turn.

Uniformity of maneuvers -- Although tests routes necessarily differ from one another, a degree of uniformity can be achieved by seeing to it that they all consist of the same number of each type of maneuver, e.g. three left turns across oncoming traffic, three with oncoming traffic controlled, and so on. Gaining uniformity is facilitated by the functional nature of the maneuvers called for. A “merge,” for example, can occur on freeway or, if one is not close enough to the license station, or at any location that requires a merge maneuver.

All routes should be driven at different times during the day with applicants in order to check on: (1) total administration time (applicants generally take longer than the person developing the test), (2) previously unnoticed travel restrictions (e.g., "No Left-Turn 4 - 6 p.m."), (3) examiner overload (too many observations), or underload (stretches where no performance can be observed), and (4) points of possible confusion in giving directions (e.g., a turn at the corner partially hidden by buildings or parked vehicles). Administrative Procedures The general administrative procedures covering road tests should be thoroughly documented.

Information to Applicants — Applicants should be apprised in advance of the performances that will be observed on the test and the criteria that will be used to evaluate them. As noted earlier, applicants should not have to take the test once to find out what skills are being tested. Publicizing the test also enhances its credibility by communicating its uniformity and objectivity. The information can appear in the driver manual or a special handout.

Communicating with applicants — Applicants should be dealt with pleasantly and courteously in order to put them at ease and thereby enable them to perform at their best. They should not, however, expect examiners to converse with them since such interaction will interfere with their performance. They should be encouraged to interpret the examiner's silence as an attempt to benefit them, not as a sign of indifference.

Giving directions — in guiding applicants during the road test, examiners should adhere to the following:

• Make instructions brief, using non-technical language;

• Employ landmarks that are obvious, and avoid depending solely upon street names,

which would not be familiar to applicants who do not live in the vicinity;

• Give the location at which a maneuver is to be made and then the maneuver (e.g., "At the next intersection, turn left"). If the maneuver is given first, applicants may respond immediately;

• Avoid including in the directions any information that is part of the test itself (e.g.,

applicants should not be instructed as to which lane to enter at an intersection). Test Support The following items are needed for support of the road testing process:

Guidance Materials — Examiners should be provided written descriptions of each route,

including the locations at which specific maneuvers are to be carried out. Examiners should expect to use route guidance materials only during practice test administrations. It normally takes but a few test administrations to commit the route and the maneuvers to memory.

Examiner preparation — Examiners should be trained in administration of the road test and monitored for their ability to provide applicant instructions, observe performance, and interpret scoring criteria. Practice should be provided by having examiners administer the test to instructors, who can make deliberate errors and who can critique the examiner's administrative procedure and scoring.

Examiner certification — each examiner should administer the test to real applicants over each test route while an instructor or senior examiner monitors the test administration from the back seat. Following the test, the instructor should review and critique the examiner's assessment. This process should continue until the instructor is satisfied that the examiner is able to administer the test correctly.

Monitoring examiner performance — the performance of all examiners should be monitored periodically. In addition, the mean scores given applicants by each examiner should be reviewed. When examiners within a single licensing station report widely divergent mean scores, their testing procedures should be reviewed to discover the source of the discrepancies.

Test Scoring The road test should be administered and scored in a manner such that each individual score reflects applicant skill to the greatest possible extent and the influence of individual examiners is minimized. Scoring System As with the knowledge test, the road test is scored by aggregating results across all the individual performances. To achieve the uniformity that characterizes knowledge tests, road testing must take account of differences among the routes over which the test is conducted. Like different forms of the knowledge test, routes can be made comparable by assuring that they involve the same number of performances and the same pattern of maneuvers. A single passing score would prevail no matter where the road test was given. In some jurisdictions, it may not be possible to render all test routes equal with respect to numbers of observations. Differences in the number of performances required can be accommodated simply by expressing the passing score in percentage terms. Then, the passing score for any one route can then be calculated by multiplying the total number of observations by the required percentage.

Recording Scores Examiners must be provided a means of recording the performance of applicants as each performance is observed. A suitable score sheet has the following characteristics:

• Every performance to be observed on the road test must appear on the score sheet, both to prompt the examiner and to permit applicant performance to be recorded. To facilitate administration and scoring, performances should be organized on the score sheet by maneuver with performances ordered in the sequence in which they occur within the maneuver, e.g. Lane Change: search signal speed etc...

• Score sheet format should provide space for examiners to record whether the applicant

passed or failed a given performance. Since most applicants will pass the majority of performances, the time required to record results can be minimized by noting only those performances an applicant has failed. Where applicants make no errors on a maneuver, an entry marked "no errors" may be checked by the examiner. Providing this entry helps distinguish truly errorless performance from instances in which the maneuver was not performed or the examiner failed to record an error.

• Space should be provided for entering the total number of correct performances or errors

(being fewer in numbers, errors are easier to count). Test forms should also include the passing score. Where the various test routes involve different numbers of performances, the passing score for each route should be either preprinted on the score sheet or provided to the examiner by some other means.

Evaluating Road Tests Road tests, like knowledge tests, can be evaluated for their reliability, the extent to which the performances making up the test accurately estimate the applicants full performance ability, validity, the extent to which the performances provide measures of the skills that they purport to measure, and effectiveness, the extent to which the road testing process improves safety of vehicle operation. Evaluating the reliability, validity and effectiveness of road test involves the same general considerations and procedures as evaluation of knowledge tests, but with some important differences resulting from differences in the nature of the tests. Road Test Reliability Just as the reliability of a knowledge test is how well it estimates and applicant’s knowledge, the reliability of a road test is a function of the extent to which the scores given to an applicant on any one administration of the test estimate the overall performance capability of applicants. Various administrations of a road test involve use of different routes, which are somewhat analogous to the different forms of a knowledge test. Each route samples the skills required in driving, just as each knowledge test form secures as sample of knowledge. However, road

testing introduces concern for another aspect of reliability that is the reliability of examiners in observing, interpreting, and recording performance. Unlike marks on a multiple choice answer sheet, the scores assigned by examiners represent subjective judgments, influenced by how much and what examiners observe and how they interpret it. The total measurement reliability of the road test is therefore a joint function of the examiner reliability and the route reliability. While it is the total reliability that counts, assessing the examiner and route reliability separately provides insight into the sources of unreliability and guides efforts to improve the test. The way reliability of road tests is generally assessed is by comparing results gained from having the test administered to the same applicant by different examiners and over different routes. If the test is measuring true performance capability, the results should be highly similar. If they are not, then the test may be measuring characteristics of the examiners and routes as much or more than it is measuring characteristics of the applicant. Similar results do not guarantee that the performance capability the test is measuring relates to safety; that is a matter of validity. Reliability is a necessary but not sufficient condition for validity. Examiner reliability The examiner reliability of road tests is estimated by having two or more examiners score applicants over the same routes (two examiners is generally the most that can see the driver well enough to score performance) and assessing the extent to which the different examiners agree on the relative scores assigned to the same applicant. It can be expressed in the same way as for knowledge tests by calculating the standard error of measurement, and making sure that it is no more than a few percentage points. In addition to assessing reliability of the entire road test, the reliability of individual performances making up the road test, such as visual search, signaling, or path, need also be examined to identify those aspects of performance with which examiners may be having difficulty. While the reliability of individual performances will be lower than those for entire tests, comparisons across performances will help point to those whose scoring criteria might be improved to raise examiner agreement. A road test can show high agreement or correlation among examiners as to who are the best performers but show substantial differences in average scores that different examiners give to all applicants. Some examiners are just more sensitive to applicants’ error than others and give generally lower scores than others, overall or with respect to specific performances. Equivalence can be assessed by comparing mean scores or pass-fail rates assigned to the same applicants by the different examiners. Unfortunately, examiners cannot be readily “equated” for difficulty by generating percentile or standard scores for different examiners in the same way as alternate test forms. Low examiner agreement and equivalence, for the overall test or at the level of individual test performances, can be addressed by seeking greater objectivity in scoring criteria and providing the instruction needed to assure that examiners can and do adhere to the scoring criteria. It is possible that certain road test performances simply cannot be dealt with in a way that yields acceptable examiner performance. In such cases it may be necessary to eliminate the

performance as an element of the road test. If examiners cannot agree upon the scoring of a performance, and there is no way of knowing who is correct, test results will not be valid and there is no point to including an invalid measure in a test, regardless of how important the behavior may be for all applicants. The fact that a road test itself is shown to be generally reliable does not mean that it will be reliably administered by all examiners in operational use. For this reason, individual license stations should monitor the overall pass rates or mean test scores for their examiners in order to identify any that are significantly above or below the average for that station. In addition to overall test results, the pass rates and average scores for each of the performances need to be examined to detect examiners who have certain “pet” performances where they are more likely to see mistakes than are others. Except for examiners who deal with special cases (e.g. elderly drivers, certain high schools) different examiners should handle the same general kinds of applicants and have similar distributions of scores. Examiners whose test results are deviant are candidates for close monitoring and possible retraining. Route Reliability Various road test routes, like different forms of a written test, can yield varying results. Tests given in city traffic can be expected to expose drivers to greater chance of error than tests in suburban areas where there is little traffic. Route reliability can be assessed at the same time as examiner reliability by having the same applicants scored by the same examiners over pairs of routes and comparing the scores given by each examiner over the different routes. The correlation across routes should be relatively high, and the standard error of measurement for estimating performance on one route from the other should be within a few percentage points. Performance on different route may correlate with one another and yet lack equivalence, owing to differences in the conditions that lead to error --- similar to differences in the difficulty of knowledge test forms or differences between “hard” and “easy” examiners. Therefore in addition to standard error measurement, the mean scores complied on the different routes should be compared. It is quite possible that substantial differences will arise within the same route over different times of day. In some location, the density of rush-hour traffic may lead to high error rates. If so, routes might be revised as to location or hours of the day. As with examiner reliability, the route reliability should also be examined at the level of individual performances to see which aspects of the test may be contributing to the error that arises. Where variation in routes leads to large errors of measurement or large mean differences, the route selection criteria need to be revised in order yield routes that are more comparable. It may prove impossible to overcome route-to-route differences for certain performances, in which case it may be necessary to eliminate the performance measure from the test in the interests of validity and equity. The fact that a road test shows acceptable route reliability in general doesn’t mean that all individual routes will give comparable results. Equivalence among examiners can be assessed in

the same way as examiner equivalence, by comparing means scores. Within each license station, mean scores compiled on the various routes should be compared with one another from time to time. If all examiners are using the same routes, differences in means scores would be expected to reflect differences in the opportunities for error. Checking for route equivalence is particularly important after changes in road characteristics, or events that might alter traffic patterns. Total Reliability The total reliability of the road test is a function of both examiner and route reliability. Where pairs of examiners give tests over two routes, it is estimated by comparing the scores given by one examiner on one route with those given by the other examiner on the other route. It corresponds most closely to the reliability of a road test in actual use, where different applicants are tested by different examiners over different routes. Again, reliability can be expressed in terms of the measurement error in estimating scores on one route with one examiner from scores on a different route with a different examiner. The inability to standardize examiners and routes to the same degree as written tests results in a reliability considerably lower than those of written tests. Road Test Validity The considerations and procedures involved in assessing the validity of road tests parallel those of knowledge tests. The methods by which performance tests are administered and scored must provide valid measures of the ability to operate an automobile in a manner that protects the safety and mobility of all road users. Applicants who lack the skills to meet these conditions should not be capable of passing the test. Like knowledge tests, the validity of road tests is not easily assessed by measuring their ability to predict who will have accidents; applicants predicted to have accidents don’t get to drive unsupervised. Their content validity can be assessed by the extent to which the performances through which they are assessed have been identified as related to safety of operation. The performances that were identified earlier derive from a systematic analysis of the performances that make up safe driving and the abilities that prior analysis of driving tasks, and research into driving skills, appear to identify as important to driving safety. Some gauge of road test validity can be gained by comparing the scores of experienced driver with those of neophytes. If one assumes that skills develop with practice, then drivers who have had a lot of experience should outscore rank beginners. In order for experience to manifest itself in skill, the experienced drivers must know the performance on which they are being scored and motivated to perform the best of their ability. Highly experienced and skillful drivers can make careless errors, which may reflect adversely upon their everyday driving but not their skills. Effectiveness of Road Tests The purpose of road tests is to prevent accidents by assuring that driver meet minimum skill

requirements before being licensed to drive unaccompanied. Their absolute effectiveness in this regard could be evaluated experimentally by comparing accidents of driver licensed with and without road tests, or some other measure of skill. In practice the effectiveness of road tests, like that of knowledge tests is unlikely to be assessed in this way due to the unwillingness of jurisdictions to issue licenses without some demonstration of at least the basic ability to drive a car. However, improvements in road testing could be evaluated through controlled experiments in which drivers licensed under competing road tests are compared for accidents and violations subsequent to licensure. Thus far no such controlled evaluation road test effectiveness appears to have been conducted. Acceptance of improved road testing procedure has stemmed primarily from the accepted validity of their content and, in some cases, their demonstrated reliability. OFF-STREET TESTING Off-street skill testing has been used primarily for three purposes: (1) initial screening for minimum skill levels before applicants are exposed to the potential hazards of road testing, (2) allowing for certain vehicle control skills to be assessed more efficiently than is possible in an uncontrolled road environment, and (3) permitting assessment of emergency skills not safely assessed in a road test. Since each of these functions involves a somewhat different array of test requirements, each will be addressed separately. Pre-test Screening The ability of applicants to control the vehicle may be so marginal as to make road testing a hazard to applicants, examiners, and the motoring public. While the number of dangerously unqualified applicants may be extremely small, the interests of safety are best served by identifying such applicants before a road test commences. This can be handled by contriving the road test to start in a parking area such that several turns and stops are required before entering the road. If a suitable parking area is not available, the test can begin on a lightly traveled side street. Assessing Vehicle Control Skills Off-street tests have been used instead of as well as in addition to road tests. The ability of applicants to handle a vehicle can be assessed more accurately in the off-road environment than on the road owing to the ability to: (1) require maneuvers that are more demanding and therefore more revealing of skill levels, (2) measure responses more precisely (e.g., stopping distance), and (3) have completely standardized test characteristics, thereby permitting collection of more uniform and reliable data. Where the necessary facilities are available, off-road testing is also economical, requiring less examiner time to obtain reliable results. Off-street skill testing is widely used in licensing automobile, motorcycle and commercial vehicle operators. Basic vehicle driving skills that are testable off-road are those involved in accelerating, (including shifting gears), braking, turning corners, and backing. Exercises capable of assessing

these skills have been devised using stanchions, traffic cones, painted lines, and in some facilities, traffic control devices such as stop sins, yield signs, and traffic lights. The shortcoming of the off-street test is its inability to measure skills involved in such traffic-related performances as merging, changing lanes, following, and judging gaps, as well as lane keeping, handling curves and approaching turns at highway speeds. As yet there is no research evaluating the relative merits of on- and off-street testing in assuring ability to drive safely. Emergency Skills Testing The off-road environment offers an opportunity to assess skill in carrying out emergency maneuvers without interference from or risk to other road users. Important emergency maneuvers include maximum braking, evasive steering, and skid recovery. None of these skills are currently part of automobile license testing. Barriers to implementation include cost of needed off-road testing equipment and facilities, potential danger to applicant, and questionable relevance to initial licensing. As yet, no research has established the relationship between such testing and safety of operation. Instruction and testing in emergency braking and swerving of motorcycles has demonstrated an accident reduction potential. While instruction in handling automobile skids has been associated with elevated accident risk in certain regions, there is no way of knowing whether it makes drivers less safe, or just encourages more driving under conditions in which such skills are required. Certainly, the value and feasibility of testing emergency skills has not been sufficiently well established to make such testing a part of licensing for automobile drivers. Evaluating Off-Street Testing Evaluating off-street testing as a mode of skill testing (not just the start of a road) imposes some special requirements.

Reliability The issue of reliability is generally less crucial to off street than on street testing for two reasons. First the high degree of objectivity in scoring minimizes the effect of differences among examiners. Second, the fact that there is just one set of performances required removes variance associated with the varying routes and traffic conditions that reduce the reliability of road tests. The variations encountered in repeated administrations of the same test are almost entirely those resulting from inconsistency of the drivers themselves. Procedures for assessing examiner and sampling reliability of off-street tests parallel those of road testing, i.e. having at least two examiners score the same drivers over at least two administrations of the test (any more than two can introduce a practice effect that masks measurement of driving skill with skill in handling the test course). Examiner and sampling reliability are measured in largely the same way as with road tests, the former by comparing the scores of two examiners during the same administration and the latter by comparing the same examiners scores across two administrations of the test. Total reliability would be the relationship between the scores obtained by one examiner on the first administration and the scores of the other examiner on the second administration. Validity The content validity of off-street testing is determined by the degree to which the maneuvers performed call for the same skills as are required on the highway. Obvious limitations are the inability to test for skills required in (1) moderate to high speed maneuvers such as merges and lane changes, (2) interacting with traffic, such as judging gaps and following distance, and (3) handling the vehicle while performing other activities, such as checking mirrors, signaling, and watching other road users. The validity off street tests will depend upon the extent to which the limited performances testable off street provide accurate estimates of the full range of on street performance. This aspect of validity can be assessed by administering both off-street and on road tests to a representative sample of applicants and comparing results that is using the road test as a criterion in evaluating the off-street test. Effectiveness As long as some measure of driving skill is to be part of the licensing process, the question of effectiveness is not whether use of an off-street skill test leads to a reduction in accidents, but whether a particular off-street test is more effective in doing so than another off-street test, or than a road test. The fact that an off-street test might not be a highly valid measure of total driving skill, as evidenced by its correlation with a road test, doesn’t necessarily mean that it is less effective in reducing accidents caused by inept driving. If it were to focus driver preparation on the skills most critical to safe operation, and measure those skills precisely, it might lead to safer driving than the more encompassing road test. Conversely, just because off-street tests correlated highly with a road test among a group of drivers doesn’t mean it will be effective in fostering the same array of skills as the road test among new drivers. In short, use of an off-street test as a substitute for a road test must be assessed through an experimental

comparison and not simply through the correlation between the two types of measures. SIMULATION The limitations of a road test as a measure of skill has stimulated interest in simulation. The potential benefits of simulation over road testing in the assessment of driving skills include scope — in a few minutes, an applicant can be confronted with an array of highway traffic situations that it might take days or weeks to encounter on the road, uniformity — every applicant can be presented with the same situation, or situations that have been equated for difficulty, automation — the examiner performance recording, scoring and debriefing functions can be carried out automatically, and safety — applicants can be presented with hazardous conditions to which examiners may be reluctant to expose to an unlicensed driver. Thus far the use of simulation has been limited to experimental applications. Despite its benefits, simulation is unlikely to serve as a substitute for a road test in licensing. First, neither the public or licensing agencies are like to accept the idea that a license should be issued or denied without some demonstration of an applicant’s ability to drive a car. Second, simulation devices involve cost whereas the road test is conducted in the applicant’s vehicle. Yet, as an adjunct to the licensing process, low-cost forms of simulation may have potential benefit in pre-screening drivers to (1) avoid testing unprepared and potentially dangerous applicants, (2) identify renewal or out of State applicants who may require road testing, (3) to guide examiners in deciding on the nature and length of road testing, and (4) to help pinpoint the source of deficiencies among driver performing poorly on the road test. Simulators appropriate to assessment of driving skills fall into two categories; interactive, in which drivers respond to simulated highway traffic scenes while the scenes change as a function of what the driver does and non-interactive, in which the simulated scenes are prerecorded and remain the same no matter what the driver does. Each form of simulation is suitable for testing different skills. Interactive Simulation An interactive simulation is one that people can actually "drive" in that simulated driving scenes change realistically as the driver operates the simulated controls. Simulated motion can be created in two ways: (1) driving a camera or optical pick-up along a 3-dimensional model of a highway environment and displaying the changing image in front of the driver, and (2) generating images by means of a computer and displaying them on a terminal. With the advances of computer technology, the latter has become the predominant type of system. The present-day interactive simulators reproduce vehicle dynamics with very high fidelity, allowing their use to teach and test for vehicle control skills — the routine skills required in accelerating, lane keeping, and braking as well as the emergency handling skills required in swerves, quick stops, and skid recovery. However, simulation truly capable of duplicating the complexity of the highway traffic environment, and changing realistically as a function of the

driver's responses, is extremely expensive. Given the economy that prevails at the time these guideline are prepared, the prospects of introducing interactive simulation into the basic license testing process are not encouraging. However, with the rate at which technology is advancing, these prospects could change, and developments bear watching. Non-Interactive Simulation In non-interactive simulation, drivers respond to recorded images of the scene ahead of the vehicle and that afforded by its mirrors. Since the images are prerecorded, the scenes do not respond to what the driver does; thus this type of simulation is unsuitable for teaching or testing vehicle control skills. Their use in license testing would be limited to testing knowledge and perceptual skill.

Knowledge -- As a knowledge testing device, simulation has the advantage over written tests of being able to get an answer without having to ask a question. While a written test can determine whether an applicant possesses certain information, it cannot assess ability to recall it when necessary; just asking the question provides a cue to recall, as does the correct answer appearing as one of the alternative responses. Simulation, like driving, can present situations requiring application of information without alerting the applicant to the need to recall information or providing cues to the information itself.

Perceptual skills -- The media that lend themselves to non-interactive simulation for licensing applications, CD-ROM, videodisc and videotape (use of film is now largely confined to group educational settings) allow a dynamic presentation of driving scenes with sufficient fidelity to assess the perceptual skills involved in judging gaps, closure rates, and in the detection of hazards. At present, these non-interactive media provide the only means of duplicating the complexity of the highway traffic environment within acceptable cost. The purpose of vehicle controls in non-interactive simulation is only to provide a means by which the drivers can register what they have perceived, such as coming off the accelerator or braking when they perceive a hazard or pulling back into a lane where passing distance is inadequate. The driver can be told to expect that scenes will not change as a function of their responses. And while, the driver's response cannot alter the driving scene, it can be designed to cause the scene to end and thus prevent conflict between what the driver does and the way the scene responds.

Because operation of the vehicle control serves simply to register a correct answer, the use of actual vehicle controls is not really necessary. License applicants can register responses by means of a joy stick, answer key, or touching the screen, thus allowing the benefits of simulation to be gained largely from the same equipment employed in automated knowledge testing. The complexity of traffic scenes that can be portrayed through video is somewhat limited by the low resolution of images capable of being presented by off-shelf video equipment. Advances in "high definition" video promise to yield substantial improvements in resolution. Evaluating Simulation

The reliability of any simulation measure would be assessed in the same way as knowledge tests, through the correlation among different samples of the performances being assessed. The validity of simulation in basic vehicle operating skills could be measured through correlation with performance of the same basic maneuvers in a real car. The knowledge and skills that involve response to stable aspects of the driving environment -- road characteristics and traffic controls -- can be validated in the same manner. In the validation process, performance in the car should be recorded with the aid of instrumentation such as video; human observation is likely to furnish a validation criterion that is less reliable than the simulation being validated. However the knowledge and perceptual skills involving other road users, such as knowing appropriate following distance or recognizing pedestrian hazards, would require conditions that can’t be stabilized in the real world. Here, content validity must be established through analyses relating the simulated conditions and required behaviors to those found in driving. The effectiveness of simulation as a part of the licensing process would have to be evaluated against the specific functions it is intended to serve. Where it is intended to add to improve the ability of the licensing process to reduce accidents, its effectiveness could be compared with that of licensing without simulation in a random experiment.



Attachment J: Sample Confidentiality & Non-Disclosure Agreement

CONFIDENTIALITY AGREEMENT

This Agreement for use and non-disclosure of confidential and proprietary information (“Agreement”)

entered into and effective this __ day of ___________, 2012 by and between:

[PROVIDER]

and

The State of Vermont

Agency of Transportation

WHEREAS, [PROVIDER] (“PROVIDER”) and The State of Vermont Agency of Transportation wish to

protect and preserve the confidential and/or proprietary nature of information disclosed or made

available to each other in connection with discussions and/or negotiations and/or an agreement

regarding services contemplated by the parties; and

WHEREAS, each party will disclose valuable, confidential, proprietary and/or inside information to the

other;

IT IS HEREBY AGREED, that neither PROVIDER nor The State of Vermont Agency of Transportation will

voluntarily disclose information for which a reasonable claim of exemption can be made, including, but

not limited to, proprietary and confidential information that is exempt from public disclosure by the

State of Vermont pursuant to 1 VSA § 317(c), subject to the following terms and conditions:


The term “proprietary information” means information, whether presented orally or in writing, and

whether or not marked as “confidential” or “proprietary,” which could provide a competitive advantage

to the party possessing such information and which either embodies trade secrets or is confidential

technical, business, or financial information provided that such information:

a. is not generally known, or is not available from other sources without obligations concerning its

confidentiality;

b. has not been made available by the owners to others without obligations concerning its confidentiality;

c. is not already available to the public without obligations concerning its confidentiality; or,

d. has not been developed independently by persons who have had no access to the information.

1. Each party agrees to use the confidential and proprietary information received from the other party only for the purposes of and in accordance with this Agreement. All proprietary rights and interests in and to a party’s confidential and proprietary information will remain such party’s property. No rights, licenses, trademarks, inventions, copyrights, or patents are implied or granted under this Agreement.

2. The receiving party shall provide at a minimum the same care to avoid disclosure or unauthorized use of the confidential and proprietary information as it provides to protect its own similar confidential and proprietary information. It is agreed that all confidential and proprietary information shall be retained by the receiving party in a secure place with access limited to only such of the receiving party’s employees or agents who have a “need to know” such confidential and proprietary information in pursuance of this Agreement.

3. All confidential and proprietary information, unless otherwise specified in writing by additional agreement, shall remain the property of the disclosing party and shall be used by the receiving party only for the purpose intended.

4. Each party agrees not to disclose the fact of their relationship with the other party to any third party

except as contemplated by this Agreement and other than as required by any law (including the Vermont Access to Records Law (1 VSA §315 et seq)), rule or regulation or judicial process.

5. The receiving party shall promptly notify the disclosing party of any request or demand by any court,

governmental agency or other person asserting a demand or request for confidential and proprietary information of the disclosing party supplied pursuant to this Agreement, so that the


disclosing party may seek an appropriate protective order. PROVIDER acknowledges that The State of Vermont Agency of Transportation is subject to the terms of the Vermont Access to Public Records Law, 1 VSA 315 et seq, and PROVIDER agrees that it will not make any claim against The State if the State makes available to the public any information it receives from PROVIDER in response to a binding order from a court or governmental body or agency compelling its production.

6. This Agreement shall remain in full force and effect for a period of two (2) years, unless earlier terminated or extended as agreed by PROVIDER and The State Agency of Transportation or protected for a longer period of time by law.

7. This Agreement may be terminated at any time during the term of the Agreement upon sixty (60) days written notice to the other party; provided that termination or expiration of this Agreement shall not relieve the recipient party of its obligations under this Agreement with respect to confidential and proprietary information exchanged prior to the effective date of the termination or expiration.

8. In the case of disputes the parties shall discuss and attempt in good faith to reach a fair, equitable and agreeable settlement. If the parties fail to settle the dispute through good faith negotiation of the issues or other forms of alternative dispute resolution, either party may terminate this Agreement immediately. This contract will be governed by the laws of the State of Vermont.

9. This Agreement may be executed in one or more counterparts (which may be originals, photocopies or copies sent by facsimile transmission), each of which counterparts shall be an original, but all of which shall constitute one and the same document.

10. This Agreement may not be amended except in writing by the parties’ authorized representatives. There are no agreements, understandings, or representations, express or implied, not specified herein.

IN WITNESS WHEREOF, the parties have caused this Agreement to be executed by their duly authorized

respective representatives as of the date first herein above written.

Provider: Vermont Agency of Transportation:

By:_______________________________ By:_______________________________

Print Name: ________________________ Print Name: ________________________

Title:______________________________ Title:______________________________


Attachment K: VAOT AFT 38

(http://www.aot.state.vt.us/conadmin/Documents/AF38revJUNE2011.pdf)

http://www.aot.state.vt.us/conadmin/Documents/ConsSpecProvNovember2011.pdf