Request for Proposal for Software Quality Assurance and

Request for Proposal for Software Quality Assurance and Validation Services for the Legacy Replacement Project Issued by: New York City Employees’ Retirement System

March 13, 2020

Software Quality Assurance and Validation Services for the LRP

RFP ID # 04282020-LRP Page i

Proposal Reference ID#: 04282020-LRP

Proposal Release Date: March 13, 2020

Dear Prospective Respondent:

The New York City Employees’ Retirement System (“NYCERS”) is initiating a Request for Proposal (“RFP”) for comprehensive Software Quality Assurance and Validation Services for the NYCERS Legacy Replacement Project (LRP).

The determination for award shall be made to the responsive and responsible proposer(s) whose proposal is determined to be the most advantageous to the New York City Employees’ Retirement System taking into consideration the price and other factors or criteria that are set forth in this RFP.


RFP ID # 04282020-LRP Page ii

Table of Contents A. Executive Summary ............................................................................................................................. 1

A.1. Project Expectations ............................................................................................................... 1 A.2. Responsiveness ..................................................................................................................... 2 A.3. Standardized Response Format ............................................................................................. 3 A.4. Definitions ............................................................................................................................... 3

B. Background and Overview ................................................................................................................... 5 B.1. LRP Governance and Risk Management ............................................................................... 6

C. Organizational Capability and Relevant Experience ............................................................................ 9 C.1. Financials ............................................................................................................................... 9 C.2. Customer History .................................................................................................................... 9 C.3. References ............................................................................................................................. 9

D. Scope of Services .............................................................................................................................. 10 D.1. Independent Testing Strategy and Plan ................................................................................ 12

Test Environments .................................................................................................. 13 D.2. Test Preparation and Execution............................................................................................ 13

Test Scripts, Cases, and Scenarios ........................................................................ 13 Testing Tools........................................................................................................... 14 Automated Testing .................................................................................................. 14 Test Evaluation ....................................................................................................... 14 Defect Management and Remediation .................................................................... 15

D.3. Reporting and Metrics ........................................................................................................... 16 D.4. Testing Support .................................................................................................................... 16 D.5. Project Team Training .......................................................................................................... 17 D.6. Knowledge Transfer .............................................................................................................. 17 D.7. Project Management of the Software QA and Validation Services Engagement .................. 18

Project Management Tools ..................................................................................... 18 Project Plan and Schedule Management ................................................................ 18 Risk and Issue Management ................................................................................... 21 Project Reporting and Metrics ................................................................................. 21

E. Project Staffing and Resource Management ..................................................................................... 22 E.1. Project Organization ............................................................................................................. 22 E.2. Respondent Project Staffing ................................................................................................. 22

Key Personnel ......................................................................................................... 24 Supporting Roles ..................................................................................................... 25

F. Proposal Components and Requirements ......................................................................................... 26 F.1. Technical Proposal ............................................................................................................... 26 F.2. Cost Proposal ....................................................................................................................... 27 F.3. RFP Exceptions .................................................................................................................... 27


RFP ID # 04282020-LRP Page iii

F.4. Additional Proposal Templates ............................................................................................. 28 F.5. Insurance Policies................................................................................................................. 28

G. Procurement Process and Requirements .......................................................................................... 30 G.1. Designated Contact .............................................................................................................. 30 G.2. Procurement Milestones and Anticipated Timings ................................................................ 30 G.3. Written Respondent Questions ............................................................................................. 31 G.4. Responses to Written Vendor Questions .............................................................................. 31 G.5. Receipt of Respondents’ Electronic and Hard Copy Proposals ............................................ 31 G.6. RFP Responsiveness and Initial Review of Written Proposals ............................................. 32 G.7. Respondent Presentations and Demonstrations ................................................................... 32 G.8. Contract Negotiations ........................................................................................................... 33 G.9. Final Evaluations .................................................................................................................. 33

H. Terms and Conditions ........................................................................................................................ 34 H.1. Multi-Term Award ................................................................................................................. 34 H.2. NYCERS Project Staffing ..................................................................................................... 34 H.3. Staff Replacements .............................................................................................................. 34 H.4. Warranty ............................................................................................................................... 35 H.5. Cost to Propose and/or Negotiate......................................................................................... 35 H.6. Collusion or Fraud ................................................................................................................ 36 H.7. Rights to Modify or Withdraw RFP ........................................................................................ 36 H.8. Travel and Other Expenses .................................................................................................. 36 H.9. Respondent Pricing .............................................................................................................. 36 H.10. NYCERS Prescribed Tools and Software ............................................................................. 36 H.11. Respondent Proprietary Tools and Software ........................................................................ 36 H.12. Ownership of Work Products and Intellectual Property ......................................................... 37 H.13. Source Code Escrow/ Software Requirements ..................................................................... 37 H.14. Agreement ............................................................................................................................ 38 H.15. Change Requests ................................................................................................................. 38 H.16. Assumptions ......................................................................................................................... 38 H.17. Subcontractors ..................................................................................................................... 39 H.18. Procurement Flexibility ......................................................................................................... 39 H.19. Termination following the Project Assessment Phase .......................................................... 39 H.20. Termination without Cause ................................................................................................... 40 H.21. Permissible Work Locations ................................................................................................. 40 H.22. Firm Fixed Price for Complete Delivery of Scope and Requirements ................................... 41 H.23. Required Deliverables .......................................................................................................... 41 H.24. Deliverable Expectations Document ..................................................................................... 41 H.25. Final Deliverable Review ...................................................................................................... 42 H.26. Payments .............................................................................................................................. 42

Project Assessment Phase ..................................................................................... 42 Phased Solution Releases ...................................................................................... 42


RFP ID # 04282020-LRP Page iv

SLA Retainage ........................................................................................................ 43 H.27. Off-Shoring ........................................................................................................................... 43 H.28. Vendor’s Acknowledgment of NYS Freedom of Information Law ......................................... 43 H.29. RFP Inconsistencies ............................................................................................................. 43 H.30. City of New York Prompt Payment Policy ............................................................................. 43 H.31. Iran Divestment Act Compliance Rider for New York City Contractors ................................. 44 H.32. (2) The City makes a determination that the goods or services are necessary for the City to

perform its functions and that, absent such an exemption, the City would be unable to obtain the goods or services for which the contract is offered. Such determination shall be made in writing and shall be a public document.Doing Business Data Form Notice .......................... 44

H.33. “Appendix A” General Provisions Governing Contracts for Consultants, Professional, Technical, Human and Client Services – Notice to Proposers .............................................. 45

I. Evaluation and Contract Award ......................................................................................................... 46 I.1. Evaluation Procedures .......................................................................................................... 46 I.2. Evaluation Criteria ................................................................................................................ 46 I.3. Basis of Award ...................................................................................................................... 46

List of Figures High-Level LRP Governance Structure .................................................................................. 6

List of Tables Table 1. Summary of Software QA and Validation Team Target Outcomes ........................................ 1 Table 2. NYCERS High-Level Statistics ............................................................................................... 5 Table 3. LRP Role Descriptions ........................................................................................................... 7 Table 4. Required Testing Types ....................................................................................................... 11 Table 5. Test Script, Case, Scenario .................................................................................................. 14 Table 6. Defect Severity Level Definitions .......................................................................................... 15 Table 7. Defect Complexity Level Definitions ..................................................................................... 15 Table 8. Defect Priority Level Definitions ........................................................................................... 16 Table 9. Blackout Periods .................................................................................................................. 20 Table 10. Respondent Proposed Staff Required Classifications .......................................................... 22 Table 11. Mandatory Key Personnel .................................................................................................... 24 Table 12. Procurement Milestones....................................................................................................... 30 Table 13. Evaluation Criteria ................................................................................................................ 46

All appendices and response templates listed below are available on the NYCERS Procurement Portal: https://nycers.procureport.com

https://nycers.procureport.com/


RFP ID # 04282020-LRP Page v

Appendices Appendix 1. Required Deliverables Appendix 2. Deliverable Expectation Document Appendix 3. Statement of Work Appendix 4. Service Level Agreements Appendix 5. Iran Divestment Act Compliance Rider For New York City Contractors Appendix 6. Doing Business Data Form Notice Appendix 7. “Appendix A” General Provisions Governing Contracts For Consultants, Professional,

Technical, Human And Client Services Appendix 8. NYCERS Supplemental Provisions Governing Software Quality Assurance and Validation

Services for Technology Projects

Response Templates Template A. Customer History Template B. Respondent Firm Customer References Template C. Technical Proposal Template D. Cost Proposal Template E. Respondent Exceptions to this RFP Template F. Vendor Information Form Template G. Vendor Acknowledgement of NY State Freedom of Information Law Template H. Acknowledgement of Addenda Template I. Bidder’s Certification of Compliance with Iran Divestment Act Template J. Doing Business Data Form Template K. Affirmations and Broker’s Forms Template L. Responsiveness Certification Form Template M. Vendor Written Questions


RFP ID # 04282020-LRP Page 1

A. Executive Summary The New York City Employees’ Retirement System (“NYCERS”) is embarking on a large-scale modernization to optimize its ability to administer retirement benefits for New York City employees and provide new and innovative services. This project is referred to as the NYCERS Legacy Replacement Project or “LRP.” NYCERS previously issued a Request for Proposal (RFP) to solicit proposals for the implementation of a pension administration solution that will satisfy LRP strategic objectives and goals.

NYCERS is issuing this RFP to solicit proposals for Software Quality Assurance (QA) and Validation services to ensure the new solution’s software code is of the highest possible quality and meets business specifications.

A summary of goals for this Software QA and Validation engagement are outlined in Table 1 below.

Table 1. Summary of Software QA and Validation Team Target Outcomes

Goal 1 Ensure High Quality Software

that Meets Business Specifications

Goal 2 Testing Knowledge Transfer to

NYCERS

Goal 3 Ensure Accountability for System

Testing

• The Software QA and Validation Team leverages expertise to validate System Integrator’s (SI’s) testing approach, methodology, and plans

• The Software QA and Validation Team conducts all necessary testing to ensure a stable, secure system that is free of defects. This testing should go above-and-beyond simply mirroring SI tests

• The Software QA and Validation Team performs additional testing after SI conducts defect remediation and system updates

• The Software QA and Validation Team ensures traceability between tests and system requirements

• The Software QA and Validation Team includes NYCERS personnel in all aspects of test planning and execution, including automated testing

• The Software QA and Validation Team ensures NYCERS personnel can effectively employ all utilized testing tools and methodology approach without assistance

• NYCERS retains all testing artifacts (e.g., test scripts, plans, etc.) post-implementation and is able to execute and build upon this library

• The Software QA and Validation Team is independently accountable for certifying that system tests are comprehensive, accurate, and complete

• The Software QA and Validation Team provides supporting evidence when issuing their recommendation for acceptance of test results

A.1. Project Expectations NYCERS expects Respondents to address the following general expectations throughout their proposal:



• Respondent’s testing approach and plan must be comprehensive and rigorous and exceed SI testing where necessary.

• Respondent’s approach and plan for system testing must exhibit significant flexibility to ensure consistency with the SI’s approach and plan for system testing based upon a phased implementation of the new pension administration solution and support the decision-making process in software acceptance.

• NYCERS will prioritize risk management throughout the project and will be supported by multiple parties. Respondents are expected to fully embrace and account for this approach to risk management in their proposal. The vendor is expected to demonstrate its commitment to coordinating with other NYCERS third-party contractors to support effective risk management activities throughout the project.

• The vendor must utilize industry-leading, commercially available, non-proprietary automated testing tools that NYCERS may adopt at the completion of the LRP.

• The vendor must utilize industry best practices in their testing approach and methodology and is required to support NYCERS’ transition in adopting this approach.

• The vendor must use NYCERS’ project management tools (e.g., Planview, see Section D.7.1) to ensure there is a clear “single source of truth” regarding all aspects of the project. These tools must provide NYCERS stakeholders with full visibility into all project activities at any time.

• The vendor should leverage relevant prior work and testing artifacts to efficiently conduct all testing activities.

A.2. Responsiveness NYCERS will only consider proposals where the Respondent (i.e., prime contractor) meets on their own without the need to rely on any subcontractors or other partners all of the responsiveness requirements. Respondents and proposals that do not meet the criteria will be deemed non-responsive. All certifications below must be provided using the Responsiveness Certification Form (Template L):

• All Respondents must have been in business for the past 10 years.

• Respondents, as a firm, must be entirely focused on independent software testing or, alternatively, have independent software testing as a core offering.

• Respondents must have provided software QA and validation services involving complex testing of ERP-like systems for at least 10 distinct organizations of equal or greater size as NYCERS (100,000 clients or more).

− Enterprise resource planning (ERP) is defined as the ability to deliver an integrated suite of business applications. ERP applications automate and support a range of administrative and operational business processes across multiple areas, including line of business, customer-facing, and administrative aspects of an enterprise. ERP deployments tend to be complex.



• Respondents must not be a contractor or subcontractor for NYCERS in another capacity relating to any other aspect of the LRP, and must not be any vendor who is submitting any proposal related to the LRP.

− The selected vendor will be precluded from pursuing or being engaged with NYCERS in capacities involving the LRP beyond software QA and validation services for the entirety of their contract with NYCERS.

NYCERS will solely determine if a proposal is responsive. Proposals deemed non-responsive will not be considered for an award (see Section G).

A.3. Standardized Response Format NYCERS requires all Respondents to use NYCERS’ prescribed templates and formats, including the main technical proposal in Microsoft Word format and provided with this RFP. Additional information about proposal components and requirements is available in Section F.

A.4. Definitions Below are definitions of some recurring terms used throughout the RFP:

• Client: an individual external User, typically an Active NYCERS Member or a Retired Client. Clients may also be individuals associated with NYCERS Members or Pensioners, such as a beneficiary.

• Employers: any City agencies or entities who employ individuals who are either eligible to join, or are required to join, NYCERS’ pension plan(s).

• NYCERS Stakeholder: any individual, internal or external, who has a personal or business interest in NYCERS as an organization and, therefore, directly or indirectly the LRP project (e.g., clients, staff, Board of Trustees).

• User: any individual, internal or external, who will utilize the LRP solution.

• NYCERS staff: individuals working at NYCERS. NYCERS at its sole discretion reserves the right to include contractors as staff.

• Respondent: the prime contractor responding to this RFP. The Respondent must be a single firm proposing to provide the solution and services described herein, either directly or through subcontractors under its direct supervision.

− The Respondent (i.e., prime contractor) may not be a partnership of firms (e.g., joint venture). In the case of multiple firms proposing to work together, one must declare itself to be the prime contractor (i.e., Respondent) with all other involved firms being formal sub-contractors to the prime contractor.

− NYCERS will hold the prime contractor solely responsible and liable for all its subcontractors. It is the prime contractor’s responsibly to ensure its agreements with subcontractors are appropriate to this end and enable full compliance with all NYCERS RFP terms and conditions and any subsequent agreements (e.g., Section H.17).



− The Respondent must on its own without consideration of any subcontractors meet all responsiveness criteria in Section A.2.



B. Background and Overview In addition to the content provided in this section, please refer to the Request for Proposal for IT Services Pension Administration Software and System Integration Services for the Legacy Replacement Project, available here: https://a856-cityrecord.nyc.gov/RequestDetail/20190401032 for additional background information on NYCERS, including: strategic vision and objectives, business divisions, pension plan offerings, and current technology landscape. Respondents should be sure to pay particular attention to Appendix 3 – Salesforce CRM Implementation Reference of the Request for Proposal for IT Services Pension Administration Software and System Integration Services for the Legacy Replacement Project. The NYCERS’ Salesforce CRM will be integral to the LRP solution and the Software Quality Assurance vendor’s scope of responsibility.

The entire Request for Proposal for IT Services Pension Administration Software and System Integration Services for the Legacy Replacement Project package, linked above, is considered part of this RFP and must be reviewed and appropriately referred to and accounted for by all Respondents in their proposal.

NYCERS was founded by the New York State Legislature in 1920 with 13,331 members. Today, NYCERS has grown into the largest municipal public employee retirement system in the United States with over 350,000 active members and retirees, including civilian employees such as clerical workers, accountants and social workers, and uniformed employees such as New York City Correction Officers and Sanitation Workers.

Table 2. NYCERS High-Level Statistics

500 Employees 11 Employers

185,000 Active Members 145,000 Retired Members and Beneficiaries

26,000 Other Clients (e.g., Vested members) Note: figures above are an approximation.

NYCERS membership is available to, and required of most, if not all, employees of the following organizations:

• City of New York

• NYC Transit Authority

• NYC Housing Authority

• NYC Health and Hospitals Corporation

• MTA Bridges and Tunnels

• NYC School Construction Authority

• NYC Housing Development Corporation

• City University of New York – Senior Colleges

https://a856-cityrecord.nyc.gov/RequestDetail/20190401032



• NYC Municipal Water Authority

For additional information, refer to page 199, Table 12 of the 2019 Comprehensive Annual Financial Report (CAFR) available here https://www.nycers.org/sites/main/files/file-attachments/cafr2019.pdf.1

B.1. LRP Governance and Risk Management Managing the risk associated with the LRP’s large and complex implementation is a top priority for NYCERS. As such, NYCERS will employ a robust governance and risk management model for the LRP that the selected vendor will be expected to fully support. This model is centered on the governance structure and risk management teams described in Figure 1 and Table 3 below:

High-Level LRP Governance Structure

1 While NYC Off-Track Betting Corporation and New York State Courts are employers listed in the 2019 CAFR, they are not listed here. This is because neither of these employers has active employees with NYCERS membership as of June 30, 2017 according to the 2019 CAFR, page 199, Table 12.

https://www.nycers.org/sites/main/files/file-attachments/cafr2019.pdf



Table 3. LRP Role Descriptions

Role Description NYCERS Board of Trustees

The Board is the governing body of NYCERS. It serves as the Fiduciary to Members, Pensioners, and Beneficiaries. Its responsibilities include oversight of the system, establishing the administrative budget of the system, promulgating rules and regulations necessary to carry out provisions of law and investing the assets of the Retirement System. In addition, the Board and the Executive Director will consider risk mitigation strategies and adoption recommendations from the Project Risk Consultant for material risks.

Project Risk Consultant Under the direction of the Board of Trustees, will have wide latitude to monitor and assess project activities and outputs for the purposes of evaluating risk throughout the project.

Executive Director The NYCERS Executive Director will be the Project Authorizer, with overall accountability for ensuring the project is appropriately resourced and helping oversee alignment with NYCERS vision and strategic objectives.

Deputy Executive Director The NYCERS Deputy Executive Director will support the Executive Director in overseeing the project at an executive level.

Chief Operating Officer The NYCERS Chief Operating Officer will be the primary executive-level manager helping oversee the project with focus on ensuring the selected Respondent delivers the project on schedule and within budget.

Independent Validation and Verification Consultant

Will advise the Executive Director, Deputy Executive Director and Chief Operating Officer, focusing on all project aspects and helping ensure the best solution is being built in the most effective and efficient way possible for NYCERS with minimal risk.

Chief Information Officer The NYCERS Chief Information Officer will be an executive-level manager helping oversee the project with focus on evaluating the degree the technical elements of the implementation are being implemented by the selected Respondent as agreed and in a manner consistent with all SOW and contractual requirements.

Enterprise Risk Committee

The Enterprise Risk Committee’s role is to oversee project and organizational risk identification, assessment, treatment (if any), monitoring, tracking and escalation of project risks.

PMO Deputy Director The NYCERS PMO Deputy Director ensures that the selected Respondent (i.e., SI) is meeting all of the project requirements and escalates issues to the NYCERS Executive Director, PMO and Chief Operating Officer. The PMO Deputy Director works closely with all the project teams to mitigate project risks and challenges including schedule, scope and budget. In addition, the PMO Deputy Director coordinates with the NYCERS stakeholders (both internal and external) to ensure all communications are disseminated in a timely manner.

NYCERS Program Management Office (PMO)

The Program Management Office (PMO) ensures all NYCERS project management requirements and policies are met and will assist in coordinating the NYCERS Project Team, as well as with any change control triggered by requests for changes (RFCs). The PMO will also work closely with all project risk management teams.

LRP Vendor(s) Responsible for the delivery of the project. Will primarily include the selected Respondent and their subcontractors, however may also include other vendors and consultants brought onto the project at the discretion of NYCERS.



Role Description NYCERS Project Team The NYCERS Project Team are the individuals who will be assigned by NYCERS to

work with the SI as well as any other vendors working on the project.

Any exceptions to this model, and the Respondent’s ability to participate in it fully, must be described in the RFP exceptions (see Section F.3).



C. Organizational Capability and Relevant Experience Each Respondent must provide an overview of its organization and demonstrate in the submitted proposal, the degree to which the organization is capable of successfully completing software quality assurance and validation on a project of the scale and complexity of the LRP.

C.1. Financials Respondents must explain the current and anticipated financial state of their organization in the submitted proposal. Each Respondent and all subcontractors (including all subcontractors of subcontractors at any level) must also submit a copy of their certified financial statements or audit reports with their proposal for each of the last five (5) years (2015–2019) in addition to any material information the expected to impact financials in the current year (2020).

C.2. Customer History In addition to providing some customer history background within their proposal, Respondents must provide a complete history of all final contracts involving public pension systems, both in the United States as well as other countries, to include both projects with an original start date on or after January 1, 2007 as well as projects ending on or after January 1, 2007 regardless of start date using Template A. All types of contracts must be disclosed (e.g., both goods and services) with a high-level description of the scope of services (e.g., Independent Testing). The contracts of subcontractors, including contracts where the subcontractor was not partnered or otherwise involved with the Respondent, must also be disclosed. If the Respondent or any subcontractors were part of any merger, pre-merger customer history with public pension systems for all involved companies must be included in the customer history.

Template A requires specific information about contracts, such as the original and current contract value, the nature of the contract, and the number of contract modifications. This type of information is generally publicly available upon request, is not confidential, and may be pro-actively published (e.g., www.checkbooknyc.com), in many jurisdictions. Respondents may neither assume that such information is confidential, nor ask NYCERS to determine confidentiality. If in specific cases Respondents believe the requested information is confidential, they must perform the due diligence to confirm confidentiality and provide the basis and rationale for confidentiality in the proposal. If NYCERS subsequently independently investigates a Respondent’s confidentiality claim and determines the information is not confidential, the Respondent’s proposal may be deemed non-responsive.

C.3. References Respondents shall provide six (6) references using Template B. These references should be for independent software quality and validation services conducted for large ERP-like system modernization projects of similar scale and complexity as NYCERS (100,000 clients or more). These references must be for customers that appear in the Respondent’s customer history (see Section C.2). NYCERS may also choose to arrange site visits directly with any or all Respondent references.

Note that for references, some specific information about contracts is requested for which Respondents must follow the requirements described in Section C.2.

http://www.checkbooknyc.com/



D. Scope of Services The vendor will be responsible for delivering all services necessary to ensure the LRP solution is of the highest quality, meets all requirements and has been designed as expected, and meets NYCERS’ expectations, in addition to other services related to testing that will support NYCERS. In their proposals, Respondents must address:

• How the proposed testing strategy is suitable and sufficient for all required testing types (see Table 4), and clearly highlighting nuances within any given testing type.

• Testing approach and methodology details, including:

− Comprehensive details on the proposed testing approach and methodology, including fully articulating how the test strategy and plan that outlines the test cycles and testing details (e.g., test coverage, techniques, traceability) will be delivered.

− The degree the proposed approach and methodology has been tailored for NYCERS and the degree it is based in proven, industry-leading practices and methodologies appropriate for projects of the scale and nature of the LRP.

− How the Respondent’s team will function independently, as well as act as an extension of the NYCERS staff, and will be directly responsible for performing independent software testing as validation of the testing performed by the SI.

− Details on all material qualitative and quantitative elements of the proposed testing approach and methodology.

− Clarity on entry and exit criteria for all testing types

− Discussion of the actual execution of testing and how it will be managed and reported on against entry and exit criteria for the testing phases

• How it will ensure that independent testing activities stay aligned with SI’s project plan:

− The SI will take a phased implementation approach for the new solution. The SI is under procurement and therefore the exact phased implementation approach is not yet confirmed (refer to the Request for Proposal for IT Services Pension Administration Software and System Integration Services for the Legacy Replacement Project for more details).

− Respondents must propose a flexible approach that can adapt as needed.

• Alignment with an Agile or Agile-like LRP SI solution implementation approach.

− NYCERS expects an Agile or Agile-like implementation to be employed by the LRP SI vendor no matter which vendor(s) are awarded.



Table 4. Required Testing Types

Testing Type Description

Integration Testing • Testing of higher-level system components as modules and other solution units are integrated by the development team (e.g., Customer Relationship Management solution).

System Testing • Comprehensive testing of the complete integrated system release.

Parallel Testing • Uses a common set of inputs to assess the compatibility of outputs from the new system with the outputs from the legacy system.

External Interface Testing • Ensures proper function of interfaces with systems owned and managed by external organization outside of NYCERS that relate to the solution (e.g., employer data exchanges).

• Includes closely coordinating and collaborating with staff of external organizations to the extent directed by NYCERS.

Performance Testing • Determines how the system (including computers, networks, software programs or devices) performs in terms of responsiveness and stability under different workloads primarily to help measure, validate or verify quality attributes such as scalability, reliability, interoperability and resource usage, as well as locate bottlenecks.

• Includes load testing and stress testing. • Verifies testing-related SI SLAs will be met.

Backup, Failover, Roll-back and Recovery Testing

• Ensures the solution meets all requirements for backup, failover, recovery, disaster recovery, and business continuity.

• Includes testing the ability to roll-back the solution from its current release to a previous release, and to legacy systems.

• Note: NYCERS may conduct backup, failover, roll-back and recovery testing independent of the project as a course of normal business. The vendor will be required to participate in such testing, including helping NYCERS to develop a strategy and plan for execution of the testing and providing feedback on test progress. The vendor may also need to provide staff augmentation during test execution.

Data Migration Testing • Ensures all functional and non-functional aspects of the application are met after migration to the new system, with data integrity and no loss of data

• Includes mock conversions and migrations

Security/Penetration Testing • Provides visibility into security vulnerabilities that could be leveraged by malicious users or others and require remedy.

• Note: Respondents should highlight the experience, skills, and certifications they possess in regards to security/penetration testing.

Regression Testing • Ensures faults have not been introduced/uncovered as a result of the changes, or introduced errors or regressions in the unchanged code.



Testing Type Description

User Acceptance Testing (UAT)

• Validation of end-to-end business processes and intended solution function from an end-user perspective.

• Note: it is planned for NYCERS staff to perform UAT under the direction of the SI. NYCERS intends to carry out a substantial UAT effort, performed by NYCERS staff at all levels in the organization for all functions and processes of the solution, end-to-end. The vendor will be required to participate in such testing, including helping NYCERS to develop a strategy and plan for execution of the testing, creating additional test suites as necessary, and providing feedback on test progress.

Post-Deployment (“Smoke”) Testing

• Ensuring the release has been successfully deployed to production and that it is behaving as designed or otherwise intended once deployed.

D.1. Independent Testing Strategy and Plan The vendor will be responsible for developing a comprehensive and detailed strategy and plan for conducting independent testing. This work will happen concurrently with LRP Project Assessment Phase (“Phase 0”) as described in the Request for Proposal for IT Services Pension Administration Software and System Integration Services for the Legacy Replacement Project (see Section B). Development of this plan must be completed during the Project Assessment Phase, which is ten (10) weeks. In their proposals, Respondents must address:

• The underlying framework that will guide development of the strategy and plan.

− Be sure to describe any industry standards (or similar) that will be used.

• The value the Respondent will provide with their services and how this will be demonstrated.

− For example, how the Respondent’s plan will provide true independent validation of system integration testing, help ensure production readiness and reduce overall project risk.

• Facilitating a series of workshops with NYCERS during project initiation to cover topics including, but not limited to:

− Determining and validating key requirements for the strategy and plan, including critical success criteria.

− Discussing and refining draft versions of the strategy and plan.

− Validating the strategy and plan.

− Discussing and validating testing environment requirements.

− Discussing and validating the plan, approach, and artifacts required to support NYCERS decision to formally accept software code.

• Aligning the progression of testing activities with the SI’s phased implementation approach.

• Approach and methodology for risk and issue identification and mitigation.



Test Environments The Respondent will propose an approach for establishing and utilizing test environments (including hardware and software) for all testing types and for all project phases. Respondents must discuss in their response:

• How they have handled the management of test environments on previous independent testing projects, highlighting the most successful strategies.

• If the approach would be to establish and maintain separate test environments from the SI what controls would be put in place to ensure uniformity between the test environments, or if the approach would be to utilize the test environments provided and configured by the SI, how the test environments would be efficiently shared between both parties to prevent any project delays.

• Any test environments that could not be hosted by a commercially available cloud provider of NYCERS choosing and why not.

D.2. Test Preparation and Execution NYCERS expects the vendor to conduct all preparation and execution of testing activities necessary to validate testing completed by the SI, as well as additional testing deemed necessary to ensure the implemented solution is stable, secure, and meets all of NYCERS business requirements.

Test Scripts, Cases, and Scenarios The execution of a comprehensive set of test scripts, cases and scenarios across all required testing types (see Table 4) is essential. The Respondent will be responsible for validating all test scripts, cases and scenarios developed and executed by the SI, as well as developing and executing additional tests necessary to ensure a stable, secure system that meets all business specifications. Respondents must provide examples of prior test scripts, cases and scenarios as well as describe the development methodology for each. Respondents must discuss their approach to ensuring accurate results are achieved from the tests. Tests are expected to be designed against an appropriate industry standard and should have characteristics such as:

• Clear association with the underlying the independent testing strategy and plan.

• Clear traceability to requirements, design, and other appropriate system elements.

• Alignment between test scripts, cases, and scenarios (see Table 4).

• Clear understanding of dependencies between tests.

• Clear identification of all inputs required to execute tests and all outputs required to verify tests (e.g., data, conditions, valid and invalid responses, etc.).

• Clear descriptions of any required technical environment components and/or configurations necessary to execute the tests.



Table 5. Test Script, Case, Scenario

Testing Term Description

Test Script • Line-by-line description of actions and data needed to perform a test

Test Case • Description of a specific idea that is to be tested without detailing the exact steps to be taken or data to be used

Test Scenario • Description of an objective a user might face when using the application

Testing Tools The Respondent must utilize enterprise-grade, industry-leading, commercially available, non-proprietary testing tools appropriate for a project of this scale and complexity, for all tests conducted during implementation of the new solution. Tools must be compatible with use of JIRA to enable streamlined defect reporting. The Respondent must grant full access to and use of tools to NYCERS, including training members of NYCERS staff to properly and effectively employ and maintain the tools, with the intent that NYCERS will adopt the testing tools after completion of the LRP. The Respondent must submit a comprehensive list of all testing tools utilized on previous projects, including the tool’s purpose and the Respondent’s level of expertise (e.g., certifications) at employing each tool. Respondents are expected to support the operation and maintenance of all utilized testing tools throughout project implementation (e.g., performing tool upgrades).

Automated Testing NYCERS wants to take full advantage of automated testing in a similar manner to other leading commercial and public organizations when implementing a solution of this scale. Regarding automated testing, all proposals must:

• Include automation of regression testing wherever such testing will be used.

• Explain if and how the automated testing will be leveraged for other types of testing beyond regression testing, and explain in which areas automation would provide the greatest benefit.

• Describe how the Respondent has utilized automated testing on prior software quality assurance and validation projects as well as the associated benefits realized over alternative methods.

• Support the proposed use of automated testing with information such as the processes and standards around automated test design and maintenance and how the proposed automated testing approach will be effective, optimal, and is aligned with the skills and level of the Respondent’s proposed resources.

• Provide details on the specific automated testing tools required to support the proposed automated approach and how they will be practically utilized during implementation.

Test Evaluation Respondents must fully articulate how they will evaluate all aspects of testing within each phase of system implementation in their proposal. All proposals must:



• Address specific steps necessary to evaluate each testing type listed in Table 3 (e.g., evaluating testing results).

• Describe the tools and processes that will be used to evaluate sufficient test coverage as well as confirm requirements traceability.

• Provide a comprehensive list of all evaluation criteria that will be utilized throughout the testing process, including entry/exit criteria and suspension/resumption criteria for each phase and type of testing.

• Outline the steps and events that would trigger situations where testing would be stalled/suspended and then subsequently resumed, including how the Respondent would help minimize potential negative impacts to the project during any such times.

Defect Management and Remediation The vendor will be responsible for validating the SI’s defect management and remediation actions, and must explain how this validation will occur, including execution of additional testing. The vendor will be required to use NYCERS’ instance of JIRA (https://www.atlassian.com/software/jira) to manage and facilitate defect reporting. JIRA must be updated as part of the defect management process and not after the fact, which must be accounted for in the Respondent’s proposal. All defects will be reported directly to NYCERS staff, not to the SI. In addition, the vendor will be required to use NYCERS’ prescribed defect severity level, complexity level and priority definitions as shown in Tables 5, 6 and 7 below.

Table 6. Defect Severity Level Definitions

Severity Level Description Severity 1: Critical

Causes complete application/solution failure or unavailability in a production environment impacting multiple users with no known workarounds acceptable to NYCERS.

Severity 2: High

Causes repeated, consistent failure of functionality impacting multiple users, or information errors/corruption, with no known workarounds acceptable to NYCERS.

Severity 3: Medium

Failure of functionality impacting one or more users but where there is a workaround acceptable to NYCERS.

Severity 4: Low

Minor failure that does not substantially affect the use of the application/solution where minimal, if any, acceptable workarounds are necessary prior to remediation (e.g., cosmetic issue).

Table 7. Defect Complexity Level Definitions

Complexity Level Description High Complex remediation required that may involve multiple solution components and/or require

senior level technical resources; and/or remediation requires 40 person-hours or more. Medium Simple remediation required that typically involves one or a very limited set of non-complex

solution components and/or junior level technical resources; and remediation requires less than 40 person-hours.

https://www.atlassian.com/software/jira



Complexity Level Description Low Simple remediation required that does not require specialized technical (e.g., programming)

skills and could be remediated by trained power users or system administrators; and remediation requires less than 10 person-hours.

Table 8. Defect Priority Level Definitions

Priority Level Description High All critical severity defects and most high severity defects. Medium High severity defects not assigned a high priority and some medium severity defects. Low Medium severity defects not assigned a medium priority and all low severity defects.

D.3. Reporting and Metrics Respondents must provide testing reporting and metrics and describe their detailed approach for doing so in their proposal, which must:

• Deliver current, on-demand testing reports, including reports that are designed to help validate the SI’s test results.

− Validation reports which must be submitted to NYCERS at the completion of each testing phase, detailing the tests conducted, the test results, and recommendations for next steps.

• Include any specific tools proposed to support testing reporting and metrics in addition to Planview (see Section D.7.1) and JIRA (see Section D.2.5).

• Use of predictive test metrics and reporting where possible.

• Explain why the proposed metrics and reporting will be effective for the LRP and what best practices the approach is based on.

• Explain how the Respondent will ensure a “single source of truth” of all their testing data (e.g., in cases where multiple tools are being used).

D.4. Testing Support Throughout the project, NYCERS may require the Respondent to perform additional services outside the scope of conducting independent testing. These may include, but are not limited to:

• Evaluating the SI’s testing strategy and plan and developing recommendations for improvement (e.g., comprehensiveness, adherence to industry best-practices, evaluation criteria)

• Evaluating the SI’s approach to testing risks and issues and developing recommendations for improvement

• Monitoring the effectiveness of the SI’s testing processes as well as the SI’s testing progress and providing feedback and recommendations for improvement



• Helping NYCERS develop a strategy and plan for NYCERS-led User Acceptance Testing and Backup, Failover, Roll-back, and Recovery Testing. The Respondent may be required to create additional test suites to facilitate various testing types, as well as provide feedback on test progress and recommendations to improve the quality and efficiency of testing efforts.

• Incorporating and facilitating meaningful participation of SI resources in specific, testing related project discussions

D.5. Project Team Training To help ensure the NYCERS staff can contribute effectively to testing-related project activities, the selected Respondent will be required to train NYCERS staff in areas including, but not limited to:

• The approach and methodology that will be employed by the selected Respondent to conduct software quality assurance, as well as the use of testing tools.

• Assisting NYCERS staff in understanding the system integrators testing approach and methodology.

NYCERS staff to be trained will be a mix of staff of various skill level including:

• Business staff, approximately 50 different individuals who will support testing activities, such as UAT, at various times throughout project implementation.

• Technical staff, approximately 50 different individuals who will support testing activities, such as review and validation of testing results, at various times throughout project implementation.

All proposed training approaches must include:

• Defining clear and measurable training objectives and providing the means and processes to objectively determine if those objectives have been met.

• Remediation and make-up training.

• Direct training of NYCERS staff as a primary training method (i.e., direct interactions between NYCERS staff and the staff of the selected Respondent, and not “train the trainer”).

− Use of training methods to complement direct training are encouraged (e.g., computer based training).

• Providing new and refresher training throughout the LRP implementation and at all critical junctures (i.e., not disproportionally providing training just once at the beginning of the project or at the beginning of the first testing phase).

D.6. Knowledge Transfer Knowledge transfer is key in NYCERS’ ability to independently conduct all types of system testing post-implementation. NYCERS expects all proposed knowledge transfer approaches to:

• Provide continuous knowledge transfer throughout the project in a way that does not disrupt project activities or otherwise risks, or causes, any project delays.



• Enable direct collaboration between the actual members of the Respondent’s team performing the work and NYCERS staff, especially those identified as being responsible for testing activities post-implementation once vendors depart (i.e., not train-the-trainer style or similar indirect models).

• Be tailored specifically to NYCERS and not generic, including creating NYCERS-specific knowledge transfer documentation, and include a process to evaluate the success of knowledge transfer using objective measures.

• Include a detailed plan that supports the transfer of knowledge and approach for determining a successful transition.

Knowledge transfer will only be deemed successful if NYCERS staff is evaluated as proficient with all aspects of planning and conducting all testing types listed in Table 3; including, but not limited to: forming a testing approach and methodology, creation of test suites (including for automated testing), test execution (including automated testing), expert use of testing tools, test evaluation and reporting, and test risk identification and mitigation strategies.

D.7. Project Management of the Software QA and Validation Services Engagement Respondents shall be responsible for providing comprehensive project management services designed to manage and ensure the successful delivery of software quality assurance and validation services. Respondents must propose a project management approach based in widely recognized best practices that is tailored to NYCERS, value-added, pro-active, collaborative, and transparent.

Project Management Tools NYCERS will require the vendor to use its instance of Planview PPM cloud-based project management software (https://www.planview.com/solutions-project-portfolio-management). Project governance, change control, and reporting will take place through Planview PPM. The required defect management tool, JIRA, is also integrated with Planview. The vendor is expected to make optimal use of capabilities enabled through that integration.

The vendor will be expected to take full advantage of all capabilities of the NYCERS’ instance of Planview, and must use Planview in a fashion that provides NYCERS with full real-time visibility into project status and all activities at all times. Respondents should clearly describe in their proposal how they will use Planview to meet these expectations.

Project Plan and Schedule Management Respondents are to provide a detailed project plan with their proposal that demonstrates the necessary buildup of resources and tasks to comprehensively fulfill all services listed within the RFP. NYCERS recognizes that as the SI procurement process is not yet complete, information such as the exact number of phased releases, as well as the scope of each release, has yet to be determined. Nevertheless, NYCERS expects Respondents to be able to include an informative project plan based on the extensive information provided in the Request for Proposal for IT Services Pension Administration Software and System Integration Services for the Legacy Replacement Project (see Section B) and based on the presumed extensive experience of the Respondent. Once a final project is agreed to with the selected

https://www.planview.com/solutions-project-portfolio-management



vendor, that vendor will be responsible for adhering to the agreed upon plan throughout the project. The project plan and explanatory information provided in the proposal should:

• Include clear work breakdown structure (WBS) information.

• Highlight key dependencies both between tasks performed by, and fully in control of, the Respondent as well as key dependencies the Respondent’s work would have with the LRP System Integration vendor.

• Provide information about how the Respondent would resource specific tasks, including estimated resource loading details (quantitative).

− Resources referenced should align with the resource types and roles the Respondent provides in their Cost Proposal.

− Respondents must be clear about what tasks are proposed to be performed on-site, off-site, or off-shore (or some combination).

• Highlight project plan connections to key milestones and required deliverables (Appendix 1).

• Ensure the proposed project plan is conservative. Proposed project plans are to account for the both the information available to the Respondent as well as known uncertainties. Proposed project plans are not to reflect “best case” or “minimum” scenarios.

− NYCERS will work with the selected vendor and the LRP SI during the Project Assessment Phase (“Phase 0”), as described in the Request for Proposal for IT Services Pension Administration Software and System Integration Services for the Legacy Replacement Project – refer to Section B, to finalize the project plan.

− While finalization of the project plan may result in redistribution of tasks, efforts, durations, etc., it may not result in an increase to the Software Quality Assurance vendor’s firm fixed price (see Section H.22). The Respondent’s project plan submitted with this proposal must be designed with this principle in mind.

• For all NYCERS led work proposed by the Respondent, Respondent must clearly identify all such tasks on their proposed project plan and ensure realistic and adequate time is afforded to such tasks, presumably informed by the Respondent’s expertise and experience. All NYCERS led work must be thoughtfully planned and designed by the Respondent, and well-supported by Respondent resources.

− All such tasks must each include specific Respondent staff/resource assignments with effort hours attached to those Respondent staff/resources to demonstrate the degree the Respondent has planned to support NYCERS led work.

− NYCERS reserves the right to require evidence from Respondents that supports the need for NYCERS led work in a particular area, as well as evidence that substantiates a Respondent’s proposed approach and estimates for NYCERS led work.



− NYCERS will not lead or perform work on unrealistic schedules, or under other unrealistic conditions, especially in any situations where the selected vendor is providing minimal support to achieve such work in the target timeframe.

• Be provided in a format where details can be easily extracted and reviewed in detail by NYCERS (e.g., MS Project or Excel). Project plans should not be provided in formats such as PDF.

In addition, the proposed project plan must account for the following blackout periods described in Table 9. During blackout periods, major deployments cannot occur. Respondents should also expect reduced availability of NYCERS staff during these times.

Table 9. Blackout Periods

Blackout Period Notes Part of January • 5th Quarter Closing

• 1099 Run (pensioners and members with taxable loans) Part of February • Mass Service Generation for Active Members Part of March–April • Annual Disclosure to active members

• Annual Disclosure to vested members All of June through the end of the first week of July

• NYCERS has significant operational commitments associated with the end of its fiscal year (June 30).

Part of September • COLA Mass Mailing Part of October • VSF Payments Last week of each Quarter • Certain files on the mainframe are closed to conduct quarterly runs

across all membership accounts.

The proposed project plan must be paired with an effective ongoing schedule management approach in order to help ensure timely delivery of the project according to the plan. Respondents must demonstrate how their schedule management approach will be pro-active and use proven methods to detect potential schedule delays and schedule risks, early, when they are more likely to be remediated.

Furthermore, NYCERS is interested in understanding, for both potential and actual schedule delay situations, what types of actions Respondents will commit to taking in response to actual schedule delays to get the project back on schedule (e.g., underestimated level of effort for a given task). Along these lines, it is important for Respondents to explain how, in these situations, they would communicate and instill confidence in NYCERS that any schedule delays, or potential schedule details, will be resolved through the Respondent’s actions. For a project of this scale, nature, and duration, it is essential that the vendor build trust and a solid track record with NYCERS of predictable resolution, and minimal uncertainty around any potential or actual schedule delays.

Respondents should anticipate and plan to resource and initiate the project within 10 days of NYCERS issuing the Purchase Order.



Risk and Issue Management Respondents are expected to pro-actively monitor for, report on, and mitigate any risks and issues within the delivery of their own software quality assurance services. Respondents are to explain in their proposal how they would plan to do this. The selected vendor will be expected to act on risks and issues in a reasonable and timely manner throughout the project.

Project Reporting and Metrics The primary purpose of project reporting and metrics is to compare actual performance to the agreed upon project plan. For example the agreed upon deliverables are furnished on time at the agreed upon level of detail and quality. Project reporting will be required to be continuous in the sense that up-to-date information should be in the Planview project management tool (see Section D.7.1) at all times, available to NYCERS stakeholders to call up in Planview on demand. Respondents must ensure their proposed approach meets this requirement.



E. Project Staffing and Resource Management Respondents must include in their proposal a comprehensive project staffing and resource management plan that meets all requirements set forth in this RFP. For example, NYCERS requires the Respondent propose specific named individuals for certain Key Personnel (see Section E.2.1). Evaluation of project staffing and resource management will include, but is not limited to, consideration of the overall proposed staffing and resource management plan, the named individuals, as well as the commitments the Respondent is willing to make regarding the minimum standards of unnamed individuals that would be assigned to the project.

E.1. Project Organization Respondents must include in their proposal a detailed project organization chart, that clearly depicts roles, assignments and subcontractors, aligns with the high-level project governance structure provided in B.1, and that describes what they believe would be most effective to successfully implement the project within the goals and expectations set forth in this RFP, including roles required of NYCERS staff, if any. Respondents must also explain the reasoning behind their proposed project organization in their proposal. This should include discussing the degree to which the Respondent has used similar project organization structures to successfully deliver other comparable projects.

E.2. Respondent Project Staffing Respondents must provide information in their proposal about every role they, and their subcontractors, will staff on the project. This includes providing information such as the role’s responsibilities, how the role is staffed, and the role’s minimum level of experience and qualifications, and assignment details. Key Personnel (see Section E.2.1) have additional requirements, such as the need to name specific individuals for all Key Personnel. In particular, NYCERS is interested in how Respondents demonstrate their level of commitment to provide quality staffing for the project, including, but not limited to, the degree to which proven individuals with demonstrated success conducting software quality assurance and validation on large scale, ERP-like system modernization projects, especially in environments similar to NYCERS (e.g., highly regulated).

Classifying Respondent Resources The staffing plan must classify each proposed key and supporting resource, both for key (named) and supporting (unnamed) roles, as described in Table 10. In order to ensure consistency across proposals, all Respondents must use these descriptions exactly as provided and may not modify or make qualifications about them in their proposal.

Table 10. Respondent Proposed Staff Required Classifications

Resource Classification Description

On-Site, Off-Site, or Off-Shore

Resources must be classified as either “on-site,” “off-site,” or “off-shore.” • On-site resources are resources that will spend 75% or more of their project working

time at a NYCERS location. On-site resources must meet the following minimum requirements:




− On-site work schedules for all resources must always be designed in the best interests of the project.

− On-site work schedules should align with NYCERS normal business hours whenever possible and where consistent with the best interests of the project.

− NYCERS may require at any time on-site resources to adjust their on-site working schedule, such as to participate in project workshops or other discussions, which will be accommodated and facilitated by the Respondent.

− On-site resources must be eligible to work in the United States for the duration of the project.

• Resources that will spend 75% or more of their project working time at locations other than NYCERS are to be classified as “off-site” if those non-NYCERS work locations are within United States or Canada, and “off-shore” if those non-NYCERS work locations are not within the Unites States or Canada. Both off-site and off-shore resources must meet the following minimum requirements: − Work performed off-site and/or off-shore must in no way hinder the project or delay

its progress. − Time-zone differences may not restrict the ability for the NYCERS team to

communicate and collaborate with off-site and/or off-shore resources during NYCERS normal business hours as needed.

− NYCERS may require at any time off-site and/or off-shore resources to appear on-site, such as to participate in project workshops or other discussions. The timing and duration of such on-site appearances requested by NYCERS will be accommodated and facilitated by the Respondent at no additional cost.

− All off-site and off-shore locations where any NYCERS-related work is performed must adhere to the minimum set of workplace standards described in Section H.21.

− In addition, no more than one-third (33.333%) of total project resources as identified by the Respondent in their cost proposal (see Section F.2) may be filled by off-shore resources.

− While Respondents are permitted to propose off-shore resourcing in a manner that conforms to the RFP, ultimately the selected Respondent’s use of any off-shore resources is solely at NYCERS discretion and only is permitted with express pre-approval from NYCERS.

Full-Time or Part-Time

Resources must be classified as either “full-time” or “part-time”: • Full-time resources are resources that will be exclusively assigned to NYCERS and will

work a minimum of 40 hours each week on the project, excluding approved holidays and personal time off, for the entire assignment period. − “Exclusively assigned to NYCERS” means a resource that will only work on the

NYCERS project and will not be assigned to, or otherwise performing work for, any other vendor client; as well as the resource not being assigned to, or otherwise performing, significant internal work for the Respondent or significant work for outside entities.




• Part-time resources are resources that will be not exclusively assigned to NYCERS (as described above) and/or will work less than 40 hours each week on the project.

• Resources that will work a minimum of 40 hours a week but that will not be exclusively assigned to NYCERS must be classified as part-time.

Key Personnel Specific individuals must be proposed for every Key Personnel listed in Table 11 below, and a résumé with three (3) references be provided for each of those individuals. In addition, in order to propose a specific individual, the Respondent must agree that the individual may not be removed from the project prior to the start date. In the event any proposed key personnel becomes unavailable, the Respondent must take the necessary action(s) to remedy in a timeframe that is acceptable to NYCERS. Furthermore, NYCERS shall be entitled to damages if the loss of key staff during the project contributes to project delay (see Section H.3).

Table 11. Mandatory Key Personnel

Role Description Test Lead The Testing Lead will be responsible for oversight and coordination of all aspects of

system testing, including coordination and oversight of all testing cycles and testing personnel. This individual will ensure that testing activities remain in sync with the SI’s testing activities, and will update the Contractor’s project manager regularly regarding the progress of testing activities. The Test Lead will provide guidance on where and how to allocate testing resources to ensure proper execution of the contractor’s testing approach, methodology, and plan. This individual will integrate NYCERS QA Team members in with the Contractor’s team in order to facilitate continuous knowledge transfer throughout the LRP. The Test Lead will confirm that all entrance and exit criteria are met within each test phase, and will ensure that all identified defects are properly communicated to the NYCERS staff. Specific experience and expertise managing complex independent testing of ERP-like systems is strongly preferred.

Project Manager The Project Manager will be the Contractor’s primary resource responsible for the management and delivery of the project, including coordinating all Contractor and Subcontractor project resources and ensuring such resources perform efficiently, optimally and produce acceptable work product at all times. The Project Manager is also responsible for ensuring continuous updating, maintenance and organization of all project documentation. The Contractor will be required to authorize and empower the Project Manager to make commitments to NYCERS relating to the project and Contractor’s scope of activities. The proposed project manager must hold one or more widely-recognized, currently active, industry certifications in project management (e.g., Project Management Professional).

Lead Test Automation Engineer

The Lead Test Automation Engineer is responsible for overseeing deployment of test automation solutions to increase the robustness, effectiveness, and efficiency of testing efforts. This individual will lead the design and execution of automated,



Role Description reusable test scripts. The Lead Test Automation Engineer will leverage the Contractor’s test automation framework in addition to best practices to determine what tests (in addition to regression testing) should be automated and in what sequence. This individual will ensure knowledge transfer of automated test execution to NYCERS staff members.

Supporting Roles Any role the Respondent proposes that is not a Key Personnel is a supporting role (e.g., Test Environment Specialist). Key information on supporting roles, such as a description of the supporting role, its responsibilities, how it will be staffed, and the minimum qualifications for the role, are required to be provided for every supporting role proposed. Respondents are not required to name specific individuals to supporting roles. However, if a Respondent chooses to name specific individuals to any given supporting roles, those individuals will be subject to the same requirements and terms that apply to named individuals in Key Personnel.



F. Proposal Components and Requirements This section describes the required components of a Respondent’s proposal. If NYCERS does not receive all required proposal components, or the components received do not meet RFP requirements, the Respondent’s proposal may be deemed non-responsive. Some portions of the proposal will require the Respondent to use NYCERS provided templates. Each template may contain additional instructions and requirements not listed below, that must be followed. There are further submissions that NYCERS must receive prior to receiving the Respondent’s proposal, such as the notice of intent to bid, which are described in Section G.

Whenever NYCERS templates are required, Respondents are responsible for fully adhering to the use and format of the templates. Respondents are responsible for independently confirming that they can open and use templates. Any questions regarding templates must be directed to NYCERS through the designated contact as part of the Question and Answer period as provided in Section G.1. Respondents are only to populate templates with the requested information and may not modify the templates to any extent for any reason. Completed templates must be submitted back to NYCERS in their original file format and not converted to any other file format, such as PDF. Similarly, attachments submitted by Respondents as part of their proposal (e.g., sample deliverables) should be in their original file format wherever possible. Proposals submitted using modified templates or the Respondent’s own templates will be deemed non-responsive.

All appendices and response templates are available at the NYCERS procurement portal at https://nycers.procureport.com/. Obtaining access to RFP materials requires registering an account on the portal, following the below steps:

1. Set your e-mail spam filter to accept e-mail from procureport.com and procureportmail.com. 2. At https://nycers.procureport.com/ click the magnifying glass icon for this RFP under “Posted

Sourcing Events”. On the page that appears click “Request Access” and follow the prompts. 3. You should receive an e-mail notification within one business day regarding access. If you do not

receive an e-mail, promptly e-mail the NYCERS designated contact (Section G.1). 4. Follow the instructions in the e-mail notification to gain access to the RFP materials.

F.1. Technical Proposal The technical proposal must include all of the following fully completed templates. Note that some templates require the Respondent to furnish certain specific attachments to accompany the completed templates (e.g., financial report):

• Template A. Customer History

• Template B. Respondent Firm Customer References

• Template C. Technical Proposal

− Must also include the complete set of attachments specified in Template C.





F.2. Cost Proposal The cost proposal must be a firm fixed price for the delivery of all RFP requirements and the Respondent’s technical proposal (e.g., services), and must be submitted using Template D. The proposed price must be adequate to accomplish the entire project scope and meet all RFP terms and requirements, with all services components aligning to the Respondent’s project plan. NYCERS may require Respondents, at any time, to detail their cost proposal basis and provide evidence that the proposed firm fixed price is reasonable for a project of this nature. Respondents will not be entitled to additional monies, special considerations or contract changes as a result of any form of underestimating or underpricing of any portion of the project scope or other costs necessary to meet all RFP terms and requirements. NYCERS reserves the right, at its sole discretion, to reduce any Respondent’s evaluation score due to any cost proposal concerns.

Furthermore, the entirety of each Respondent’s cost proposal will be the requested information populated on Template D. Respondents may not include any supplemental or unsolicited information with their cost proposal. Including any such information with the cost proposal may result in a non-responsive determination.

F.3. RFP Exceptions The vendor will be contractually bound by all aspects of this entire RFP package, without negotiation, except where RFP exceptions were provided with the Respondent’s proposal using Template E. RFP exceptions are any specific aspects or components of the entire RFP package, including attachments, addendums, ancillary documents, any aspects of the RFP marked “sample” or “example,” etc., which a Respondent will not agree to be contractually bound to with NYCERS, as published, as a result of this procurement.

All Respondents must submit a completed Template E with their proposal that clearly and specifically describes all exceptions with the RFP as defined above or alternatively states they have no exceptions. NYCERS will not entertain RFP exceptions made after proposals are received, and only those exceptions identified in Template E will be considered by NYCERS. Any exceptions found elsewhere in the proposal will be superseded by the provisions of this RFP and will not be considered or honored by NYCERS and may lead to a non-responsive determination. Failure to cite exceptions constitutes full acceptance of the RFP elements, regardless of whether the Respondent’s omission was unintentional. In addition, RFP exceptions are not an opportunity for Respondents to communicate preferences for alternatives for any given aspect of the RFP package where the Respondent would ultimately accept the original published content.

All RFP exceptions provided must meet the following requirements:

• Each exception must be standalone and not contingent or dependent on any other exception in any way.

• For each exception an alternative must be proposed that would be acceptable to the Respondent and that the Respondent reasonably believes will be acceptable to NYCERS.

− NYCERS reserves the right to accept any exception alternative proposed without further negotiation or any opportunities for revision.



• All information provided regarding exceptions must be specific and precise. Generalizations, deferment, “hedging” and similar language are not permitted.

NYCERS will review each exception and make one of the following general determinations at its sole discretion:

• To accept the exception and proposed alternatives as submitted by the Respondent.

• To reject the exception and proposed alternatives as submitted by the Respondent as unacceptable to NYCERS and deem the proposal non-responsive.

• To opt to discuss or negotiate the exception and proposed alternatives with the Respondent at a future stage of the procurement process (e.g., contract negotiations).

− NYCERS will not accept as a result of contract negotiation any terms less favorable to NYCERS than the proposed alternatives submitted by the Respondent in their written proposal, as solely determined by NYCERS.

NYCERS may choose, at any time, and at its sole discretion, to change its determination on any RFP exception. For example, for any exception NYCERS is willing to negotiate, cease negotiations and either accept an alternative originally in the Respondent’s proposal without modification or negotiation, or to reject the exception and any proposed alternatives and potentially as a result deem the proposal non-responsive and end negotiations. A Respondent’s evaluation score may also be impacted as a result of RFP exceptions as deemed appropriate by NYCERS.

F.4. Additional Proposal Templates NYCERS requires the following additional templates to be completed and received simultaneously with Respondent proposals:

• Template F. Vendor Information Form

• Template G. Vendor Acknowledgement of NY State Freedom of Information Law

• Template H. Acknowledgement of Addenda

• Template I. Bidder’s Certification of Compliance with Iran Divestment Act

• Template J Doing Business Data Form

• Template K. Affirmations and Broker’s Forms

• Template L. Responsiveness Certification Form

F.5. Insurance Policies In addition to the Certificates of Insurance and Certification of Insurance Broker or Agent (Template K), Respondents shall provide copies of insurance policies for the following insurance types listed in Schedule A of City Appendix A: 1) Workers’ Compensation, 2) Disability Benefits Insurance, 3) Employers’ Liability, 4) Commercial General Liability, 5) Professional Liability/Errors & Omissions, 6) Crime Insurance, and 7) Cyber Liability Insurance. Respondents must use the Insurance Policy Checklist, included in Template K,



to indicate which insurance policies are being submitted and any types of insurance the Respondent does not hold. If any provided insurance policies are deemed by the Respondent to be confidential, Respondent must so indicate in Template G, Vendor Acknowledgement of NY State Freedom of Information Law.



G. Procurement Process and Requirements One requirement for Respondents to be eligible to be considered responsive (see Section A.2), is meeting all procurement process requirements (e.g., deadlines) and participating in all aspects of the procurement process as described in this RFP. Any Respondent who does not meet procurement requirements or adhere in good faith to the prescribed procurement process may be deemed non-responsive. Note that NYCERS expects to narrow down the field of Respondents as it moves through the procurement process described in this section and may not invite all Respondents to participate in later stages of the process such as respondent presentations and contract negotiations. NYCERS also reserves the right to modify or cancel the procurement process at any time, for any reason.

G.1. Designated Contact All inquiries to NYCERS regarding this procurement, or anything potentially related to this procurement, are only permitted to be made in writing via e-mail to the designated NYCERS contact below. Inquiries to NYCERS made outside of the designated contact may disqualify the Respondent and any other related parties from the procurement at NYCERS sole discretion.

Contact: Tenaye Wolde E-Mail: [email protected]

G.2. Procurement Milestones and Anticipated Timings Table 12 summarizes procurement milestones and anticipated timings. Respondents are required to adhere to the stated timings and deadlines unless amended by NYCERS. NYCERS does not guarantee the milestones and timings presented will remain unchanged and may amend the milestones and timings at its sole discretion at any time. Additional information regarding the milestones listed in Table 12 are included in this section. All question submissions and correspondence are to be made via e-mail to the designated contact (see Section G.1) unless otherwise specified.

All other RFP related Appendices, Forms, and Templates must be accessed and submitted through the NYCERS Procurement Portal via this link: https://nycers.procureport.com/

Where a specific NYCERS prescribed template is required, any partially completed or modified templates received by NYCERS (e.g., legal language changes), and/or that include any additional unrequested documentation (e.g., riders), will not be accepted.

Table 12. Procurement Milestones

Milestone Description Timeframe or Deadline RFP Published in City Record Online March 13, 2020 Written Respondent Questions Due Must be received by NYCERS no later than

March 24, 2020 5:00 PM EST NYCERS’ Response to Questions published in City Record Online

March 31, 2020 (estimated)

Respondent Electronic and Hard Copy Proposal Submission Deadline

April 28, 2020 5:00 PM EST

mailto:[email protected]




Milestone Description Timeframe or Deadline Proposal Responsiveness Review Period April – May 2020 (estimated) Initial Evaluation Committee Review of Written Proposals May – June 2020 (estimated) Respondent Presentations and Demonstrations Note: NYCERS may not invite all Respondents to participate in Presentations and Demonstrations Notice of Award and Project Start January 2021 (estimated)

G.3. Written Respondent Questions NYCERS designated contact must receive from all Respondents their written questions by the deadline specified in Table 12. Even if a Respondent has no specific questions, Respondents must send an e-mail to the designated contact to that effect that must be received by the deadline specified in Table 12. Respondents are hereby notified this is the only fully open opportunity to ask material questions or request clarifications regarding the RFP.

G.4. Responses to Written Vendor Questions Responses to all written questions received will be made available to all Respondents electronically on The City Record Online.

G.5. Receipt of Respondents’ Electronic and Hard Copy Proposals Proposals must be received by NYCERS by the deadline in Table 12 in two formats as described below. Proposals received after the deadline will be deemed non-responsive. However, Respondents will be afforded an opportunity to file an appeal pursuant to the NYC Procurement Policy Board (PPB) Rules. Pursuant to New York City’s Procurement Policy Board Rules, Vendors have the right to appeal NYCERS non-responsiveness determinations within five (5) days and NYCERS non-responsibility determinations within ten (10) days and to protest NYCERS’ determination regarding the solicitation or award of a contract in writing to NYCERS, 335 Adams Street, Suite 2300, Brooklyn, NY 11201.

The determination of appeals by NYCERS will be made in writing and are final. Any administrative RFP process questions about proposal submission should be submitted to the designated contact as soon as possible (such questions are not subject to the written questions process described in Section G.3).

• Electronic copy – an electronic copy of the proposal, including all templates and attachments, must be uploaded to NYCERS procurement portal: https://nycers.procureport.com/.

− Instructions on gaining procurement portal access are provided in Section F.

− Respondents are expected to take the time to closely follow the instructions within this RFP and as shown on the NYCERS procurement portal or the Respondent’s proposal may be deemed non-responsive.

− Respondents are solely responsible for allowing adequate time to properly upload their proposal before the submission deadline in Table 12, including ensuring sufficient time to respond to unforeseen circumstances, such as an interruption in the Respondent’s internet connection that prevents or degrades access to the NYCERS procurement portal.




• Hard copies – eight (8) exact ring-bound paper copies of the electronic proposal, including all templates and attachments, must be received by NYCERS no later than the proposal submission deadline in Table 12.

− Cost proposal copies must be provided in a separate sealed envelope.

− For any hard copies sent via mail or carrier, Respondents, not the postal service or selected carrier, are solely responsible for ensuring adequate time to help ensure NYCERS receives the hard copy proposals in advance of the submission deadline in Table 12.

o Respondents are strongly encouraged to purchase appropriate proof of delivery services.

− Hard copies may alternatively be delivered in person by the Respondent firm. Respondents are solely responsible for ensuring adequate time to help ensure NYCERS receives the hard copy proposals in advance of the submission deadline in Table 12, including, but not limited to, accounting for potential travel delays and allocating time to go through building security.

− All hard copies are to be received at NYCERS executive offices (below). In person deliveries by carriers or the Respondent must check-in at NYCERS reception on the 22nd floor.

New York City Employees’ Retirement System Attn: Tenaye Wolde 335 Adams Street, Suite 2300 Brooklyn, NY 11201-3724

The New York City Comptroller is charged with the audit of the contract in New York City. Any Vendor who believes that there has been unfairness, favoritism or impropriety in the proposal process should inform the Comptroller, Office of Contract Administration, 1 Centre Street, Room 1005, New York, NY 10007; telephone number (212) 669-2323.

G.6. RFP Responsiveness and Initial Review of Written Proposals For all proposals received, NYCERS will first conduct a review to determine if the proposal is responsive (see Section A.2). Responsiveness will solely be determined by NYCERS. Proposals deemed non-responsive will not be further considered. Should a proposal be deemed non-responsive, Respondent will be afforded an opportunity to file an appeal pursuant to the NYC Procurement Police Board (PPB) Rules. The determination of appeals may by NYCERS will be made in writing and will be final. Only proposals deemed responsive by NYCERS will be evaluated (per the criteria set forth in Section I). Please note that NYCERS reserves the right to award the contract on the basis of the initial proposals received, without soliciting a best and final offer (BAFO), further discussions or negotiations with the Respondent. Therefore, it is important each Respondent’s initial proposal should contain its best terms.

G.7. Respondent Presentations and Demonstrations After the initial review of proposals, NYCERS anticipates inviting some, but not all, Respondents to deliver an in-person presentation to NYCERS as well as to participate in demonstration exercises. In order to be considered for selection, Respondents invited must accept NYCERS’ invitation and in good faith fully participate in this stage of the procurement process as directed by NYCERS. Regarding in-person presentations, NYCERS anticipates issuing guidelines Respondents must adhere to (e.g., timing



limitations) in advance of scheduled presentations. Demonstration exercises are anticipated to be some combination of delivery team interactions with NYCERS staff as well as demonstrating how proposed testing tools would be used during project implementation. NYCERS plans to provide specific details when distributing invitations to Respondents.

G.8. Contract Negotiations If NYCERS chooses to negotiate with a Respondent, in order to be considered for selection, the Respondent invited must accept NYCERS’ invitation and fully participate in this stage of the procurement process as directed by NYCERS. The scope and timeframe of contract negotiations will be solely determined by NYCERS. NYCERS is under no obligation to move forward with any Respondent(s) it chooses to negotiate with.

G.9. Final Evaluations NYCERS will finalize evaluations and subsequently determine if any contracts will be awarded. NYCERS is under no obligation to award any contracts at the end of the procurement process. Should NYCERS decide not to contract with the Vendor deemed most responsible and responsive, it in its sole discretion may negotiate with the next Vendor deemed most responsible and responsive.



H. Terms and Conditions Submission of a response or a proposal shall constitute the Respondent’s acknowledgement it has fully reviewed the entire RFP package, and fully accepts the terms and conditions herein, including Appendix A and Appendix A1 barring any the specific exceptions provided by the Respondent with its proposal (see Section F.3).

H.1. Multi-Term Award NYCERS intends to award this as a multi-term contract (more than one year). The multi-term contract is subject to modification or cancellation if adequate funds are not appropriated to NYCERS to support continuation of performance in any fiscal year succeeding the first. The contract is subject to modification or cancellation if the selected Respondent’s performance is not satisfactory. The NYCERS Contracting Officer shall notify the selected Respondent as soon as is practicable if the funds are, or are not, available for the continuation of the multi-term contract for each succeeding fiscal year.

H.2. NYCERS Project Staffing NYCERS does not guarantee assignment of personnel or other resources, including both individuals directly employed by NYCERS and any contractors NYCERS may employ outside of the selected Respondent, to the project, and will not enter into a binding commitment with the Respondent regarding NYCERS assigned resources (e.g., time, number, skill, experience). Any formal or informal information on NYCERS staff resources and commitment level to the project shall in all instances be considered estimates and shall not be interpreted under any circumstances as a binding commitment by NYCERS, including when such information is provided in contract documentation. NYCERS reserves the right to change, reduce, or increase NYCERS assigned resources at any time, for any reason, and without advance notice to the Respondent, before and during the project. Respondents are exclusively responsible for delivery of the project within the agreed upon timetable, including where delivery is being provided by the subcontractors of the selected Respondent. As such, every Respondent’s proposed staffing plan must be designed with the assumption that NYCERS resources may not be available as the selected Respondent expects or planned for. The selected Respondent will not be entitled to special considerations (e.g., contract modifications, deadline adjustments, additional monies) due to NYCERS project staffing.

H.3. Staff Replacements NYCERS, at its sole discretion, reserves the right to remove any of the selected Respondent’s staff or subcontractor’s staff from the project, for any reason and without advance notice. Upon written notification, the selected Respondent shall have no more than ten (10) business days to furnish NYCERS with resumes for a minimum of three (3) replacement candidates with equal or greater skills and experience as the removed staff member, and two (2) references for each proposed replacement candidate. NYCERS may require an in-person interview for any or all proposed replacement candidates, and the selected Respondent will ensure that the replacement candidate(s) can be onsite for an interview within ten (10) business days of NYCERS’ request. The selected Respondent shall also ensure that the replacement candidate is available to start within 15 business days of NYCERS’ candidate selection. In the event none of the initial proposed replacement candidates are acceptable to NYCERS, NYCERS reserves the right to require the selected Respondent to furnish additional resumes indefinitely until an acceptable candidate is selected.



Unless mutually agreed upon by both parties, the selected Respondent and its subcontractors must maintain composition of the project team at the agreed upon levels throughout the project timeline. Respondent must receive express written consent from NYCERS before removing staff members from the project, reducing staff member involvement, reassigning staff members, replacing staff members, or taking any similar staffing actions. Exceptions will only be considered for staffing situations outside the control of the selected Respondent (e.g., staff member resigns from the selected Respondent’s firm). In these events, NYCERS must be notified within three (3) days of such event out of Respondent’s control, and the staffing replacement process and requirements described above will apply. NYCERS may require proof of such situations from the selected Respondent.

NYCERS shall be entitled to seek damages if any of the selected Respondent’s staffing losses and/or replacements contribute to project delay or degradation of quality work products.

H.4. Warranty Warranty will begin upon the first end-user solution deployment and will end 12 months from NYCERS’ final acceptance of all project deliverables after full implementation. Defects discovered near the end of the warranty period shall automatically extend the warranty period by 90 days past the date the last defect was resolved unless otherwise agreed by NYCERS. Similar automatic extensions shall apply to defects discovered during the extended warranty period. All automatic extensions shall be provided at no additional cost to NYCERS. NYCERS shall notify Respondent in writing of the automatic extension. Notwithstanding the foregoing, Respondent is responsible for the extension period and a failure to provide notification does not constitute a waiver. In addition, NYCERS shall be entitled to purchase from the vendor an extended warranty for any number of additional months up to three (3) years, commencing on the day the original warranty ends (after any automatic extensions), for identical terms, coverage, and levels of service as the original warranty (e.g., including the terms that will automatically extend the warranty beyond its original term based on a qualifying event). In the event NYCERS does not purchase the maximum possible extension, NYCERS reserves the right to purchase further extensions, at the original agreed upon price, up to the maximum at a later time.

All proposed warranties must be managed directly by the vendor, and not by a subcontractor, and must cover the entirety of the solution, including customizations. The warranty shall not be voided, in part or in whole, by any supported changes to the solution made by NYCERS’ staff post-deployment (e.g., defects discovered after actions such as modifying business rules or workflow, base product upgrades, enabling and configuring new features or other capabilities supported by the product), and such post-deployment changes shall be fully covered by the warranty. The warranty must also meet all SLAs as described in Appendix 5.

H.5. Cost to Propose and/or Negotiate All costs directly or indirectly incurred in reviewing, responding to, and or participating in this RFP, shall be solely borne by the Respondent. NYCERS shall not be liable for any costs or expenses incurred by Respondents as a result of this RFP, the RFP procurement process, NYCERS’ modification or withdrawal of this RFP, NYCERS’ determination that a contract shall not be awarded, and/or any costs associated with the negotiation of the Agreement.



H.6. Collusion or Fraud By submitting a proposal, each Respondent represents and warrants that its proposal is not made in connection with any other Respondent that is submitting a response to this RFP, and that its proposal is in all respects fair and without collusion or fraud. Any evidence of agreement or collusion among Respondents to illegally restrain freedom from competition, such as by agreement to offer a fixed price or otherwise, will disqualify all involved entities (e.g. Respondent, subcontractor) from this procurement and any related procurements.

H.7. Rights to Modify or Withdraw RFP NYCERS reserves the right to modify or withdraw this RFP at any time. Any notice or amendments by NYCERS to this RFP shall be set forth in writing, and sent to all prospective Vendors known to have received this RFP and/or published on the City Record online. Such amendments shall be distributed within a reasonable time to allow prospective Vendors to consider them in preparing their proposals.

H.8. Travel and Other Expenses Respondents shall build in travel time, travel costs, and any other appropriate expenses within its Cost Proposal. NYCERS shall not be liable for any additional monies resulting from an underestimation of, or unanticipated travel and other expenses for the entire duration of the project. Unless otherwise specified by NYCERS, each travel expense will not be independently billed. Respondents are not permitted to directly or indirectly attribute time or costs for expenses to the project, such as: travel time, product or job function training, time to complete administrative items necessary to meet the selected Respondent’s internal firm requirements.

H.9. Respondent Pricing Respondents are hereby notified that, unless the proposal is withdrawn, the prices stated in their proposals and BAFO’s are final and valid until contract award. Offers may be withdrawn in writing to NYCERS before an award is made. NYCERS reserves the right to request a cost breakdown of the Respondent’s proposed price, including details above and beyond what is required in the Cost Proposal template (Template E). Respondents are not permitted to apply a direct or indirect markup of any kind for any third-party products or services proposed that may ultimately be procured by NYCERS through contracts with the selected Respondent, including but not limited to third-party software licenses and maintenance costs. NYCERS reserves the right directly procure third-party software licenses and maintenance services.

H.10. NYCERS Prescribed Tools and Software NYCERS may request the selected Respondent use specific software or tools to support project implementation, or may change the tools or software used to support the project at any time during the course of the project. The selected Respondent will comply with such requests at no additional cost to NYCERS and without causing negative impacts to the project (e.g., delays).

H.11. Respondent Proprietary Tools and Software Where the selected Respondent is permitted by NYCERS to use proprietary tools or software, the selected Respondent agrees to provide full access and transparency into such tools or software to the extent requested by NYCERS.



H.12. Ownership of Work Products and Intellectual Property NYCERS will have all ownership rights to hardware and software, or modifications thereof, as well as associated documentation designed, developed, or installed. All data, technical information, materials gathered, originated, developed, prepared, used, or obtained in the performance of the contract, including, but not limited to, all reports, surveys, plans, charts, literature, brochures, mailings, recordings (video and/or audio), pictures, drawings, analyses, graphic representations, software computer programs and accompanying documentation and print-outs, notes and memoranda, written procedures and documents, regardless of the state of completion, that are prepared for or are a result of the services required under this contract shall be and remain the property of NYCERS and shall be delivered to NYCERS upon 30 days’ notice by NYCERS. With respect to software computer programs and/or source codes developed for NYCERS, the work shall be considered “work for hire,” i.e., NYCERS, not the Respondent or subcontractor, shall have full and complete ownership of all software computer programs and/or source codes developed.

All work products and deliverables produced under contracts awarded as a result of this proposal will be the exclusive property of NYCERS. This includes, but is not limited to, software, test scripts (manual, automated), documentation, and development materials. Respondents shall not sell a work product or deliverable produced under a contract awarded as a result of proposals without explicit permission from NYCERS.

NYCERS’ ownership interest in both the solution and the source code delivered, for the base components of the solution, the customized elements of the solution, and the enhancements provided to each through post-implementation, out year support, is a license to use them in perpetuity.

The Respondent’s proposal must clearly describe the terms of all licensing considerations, such as an End User License Agreement.

H.13. Source Code Escrow/ Software Requirements The vendor’s proposed solution must NOT include software or hardware locks, traps, dongle keys, or similar security measures that would in any way deny NYCERS full and complete access.

NYCERS requires a nonexclusive, unlimited perpetual license to software for source code, as well as associated documentation designed, developed, or installed. Such nonexclusive license shall apply to all data, technical information, materials gathered, originated, developed, prepared, used, or obtained in the performance of activities associated with the procurement. Additionally, the scope of this license shall include, but not be limited to, all reports, surveys, plans, charts, literature, brochures, mailings, recordings (video and/or audio), pictures, drawings, analyses, graphic representations, software computer programs and accompanying documentation and print-outs, notes and memoranda, written procedures and documents, regardless of the state of completion, that are prepared for or are a result of the services required and to be delivered under this procurement.

The source code for any software (be it the customized line-of-business application, middleware, a code generator, a specialized I/O routine, or any similar or related item) which is developed by the Respondent, or an affiliate company that is 20% or more owned by the Respondent, and used in the new system must be delivered to NYCERS. The Respondent must agree to deliver application source code with the delivery of each functional rollout phase during the project. NYCERS’ acceptance of a phase will be contingent upon this requirement being met. In addition, once the first functional rollout phase has been delivered,



source code updates must be delivered no less frequently than quarterly. Source code for linkages to/from the system (which may be developed by a ‘sub-contractor) is similarly to be provided. It is to be updated and delivered to NYCERS throughout the duration of the project.

In addition, the source code for any third party software that is delivered and/or licensed to NYCERS as part of the new solution must be escrowed on NYCERS’ behalf if, at the time of final turnover and acceptance of the new system to NYCERS, that software product has ever previously been escrowed for the benefit of any other client of the third party provider.

H.14. Agreement In the event that NYCERS chooses to proceed with a Respondent, the proposed Agreement that NYCERS anticipates to award will be based on the proposal submitted by the chosen Respondent and specifications of this RFP. The proposed Agreement between NYCERS and the vendor shall be a combination of the specifications, terms, and conditions of this RFP, including the terms included in any documents attached to the RFP, the Respondent’s technical and cost proposals, any written clarifications or changes made in accordance with the provisions set forth within this RFP, and any other terms deemed necessary by NYCERS or negotiated between the parties. NYCERS reserves the right to terminate all contracts for any reason at any time without penalty. By submitting a proposal, each vendor acknowledges its acceptance of these specifications, terms and conditions without change except as otherwise specified as RFP exceptions (see Section F.3).

H.15. Change Requests Change requests will be exclusively managed via the change control provisions established in the agreement with the selected Respondent. NYCERS may, at its sole discretion, require the selected Respondent to provide information, documentation and/or evidence as a condition of considering a change request. For example, evidence that the selected Respondent provided NYCERS with advance notice of a potential change request and explored reasonable alternatives to a change request may be required. The selected Respondent will not be entitled to additional monies related to change requests (e.g., preparing, researching). Moreover, the selected Respondent shall not recover retroactively for costs incurred prior to the contractual execution of a NYCERS approved change request. All proposed cost increases associated with proposed change requests will require detailed information as determined by NYCERS, which at minimum shall include a rationale as to why the cost increase is permissible to be sought per the contract, an accounting of all actions taken to avoid the need for a cost increase, and a detailed buildup of the proposed cost increase.

H.16. Assumptions To help ensure all Respondents’ proposals are based on the same set of RFP details and other foundational criteria, Respondents are not permitted to make their own assumptions in their technical and cost proposal, or with regard to any activity related to their proposal or this procurement (e.g., oral presentations, contract negotiations). By submitting a proposal, Respondents certify they have requested clarification from NYCERS on all elements of the RFP during the written question period (see Section G.3) and have updated their entire proposal submission (e.g., technical, cost) accordingly based on any NYCERS’ responses. Any assumptions found in proposals will be disregarded by NYCERS without an opportunity for the Respondent to revise their proposal. Respondents are reminded of the option to submit



exceptions to this RFP as they see fit (see Section F.3). Furthermore, Respondents hereby agree that assumptions, or any equivalent, will not be added or otherwise included in any subsequent contract documentation that may be agreed to with NYCERS.

H.17. Subcontractors NYCERS will hold the prime contractor solely responsible and liable for all its subcontractors. NYCERS requires full visibility into all firms involved in the project. Therefore, the selected Respondent may only employ subcontractors with the express written approval of NYCERS. Similarly, subcontractors, at any level, may only employ subcontractors with the express written approval of NYCERS. The selected Respondent is solely and fully responsible and accountable for all work performed by subcontractors, including full costs of remediating any inadequate work products; including the event that NYCERS, for any reason, requires subcontractors be removed from the project. The selected Respondent will warrant that subcontractors comply with all terms and conditions in this RFP and any resulting Agreements. The selected Respondent’s formal agreements and any informal arrangements with subcontractors must:

• Not place any restrictions or limitations on subcontractors from contracting separately with NYCERS at any time for any reason.

• Permit subcontractors to interact directly with NYCERS, including outside the presence of any selected Respondent staff.

• Require subcontractors to be fully transparent with NYCERS regarding their activities, work product and any other project elements.

• Not penalize or restrict in any way subcontractors from directly communicating to NYCERS project concerns, risk or other information material to the success of the project, including any situations that, in the selected Respondent’s opinion, may not create a favorable perception of the selected Respondent to NYCERS.

H.18. Procurement Flexibility NYCERS, at its sole discretion, may purchase any components/elements and/or services (e.g., hardware, software, cloud services) for this project by means of any procurement vehicle(s) of its choosing, and is not obligated to purchase any components/elements and/or services from the selected Respondent. NYCERS shall not owe the selected Respondent monies or other compensation for exercising its discretion to make such purchases outside the contract with the selected Respondent.

H.19. Termination following the Project Assessment Phase Respondents are advised that NYCERS expects to complete the effort with a single Respondent. However, the proposals and ensuing Agreements shall provide that at the end of the Project Assessment Phase (i.e., Phase 0), NYCERS may, at its sole discretion and with no penalty, terminate the effort at any time including not moving forward with the Respondent to a subsequent phase. NYCERS liability will be limited to payment of an amount equal or less than the costs set forth in the payment structure set up based on deliverables and accepted by NYCERS. NYCERS shall not be liable for any other cost, overhead, profits, or damages.

This section does not apply to termination in the event of a material breach of contract.



Should NYCERS elect to terminate the project at the end of the Project Assessment Phase, the termination shall be subject to the following terms and conditions:

• Termination shall become effective upon NYCERS’ sending a written or electronic notice to the Respondent.

• Termination shall not be considered for cause or for default, and notice of termination shall not constitute an admission or accusation of any wrongdoing on the Respondent’s part in connection with the termination.

• Upon termination, NYCERS and the Respondent shall each execute a covenant not to sue and a general release of all claims or potential claims, including those for costs, profits, liabilities, or damages, as of the date of termination. However, NYCERS may assert a claim that Respondent failed to return NYCERS materials and data, which shall include all original files and other materials, and all copies, including unauthorized copies, of such materials in all media, including portions of either originals or copies, in Respondent’s possession or under its control. Respondent shall agree that it will refrain from using any such materials or copies thereof. NYCERS’ materials consist of but are not limited to devices, records, data, notes, reports, proposals, business cards, letters, specifications, drawings, equipment, and other materials. Respondent shall agree to erase or destroy any NYCERS materials and portions thereof contained in all types of computer memory, and so warrant in writing to NYCERS within thirty (30) days of termination.

H.20. Termination without Cause Termination shall become effective upon NYCERS’ sending a written or electronic notice to the Respondent. NYCERS shall only be liable for payment of an amount equal to the costs allocated to the phase that was completed and accepted by NYCERS; NYCERS shall not be liable for any other cost, overhead, profits, or damages.

Termination shall not be considered for cause or for default, and notice of termination shall not constitute an admission or accusation of any wrongdoing on the Respondent’s part in connection with the termination.

Upon termination, NYCERS and the Respondent shall each execute a covenant not to sue and a general release of all claims or potential claims, including those for costs, profits, liabilities, or damages, as of the date of termination. However, NYCERS may assert a claim that Respondent failed to return NYCERS materials and data, which shall include all original files and other materials, and all copies, including unauthorized copies, of such materials in all media, including portions of either originals or copies, in Respondent’s possession or under its control. Respondent shall agree that it will refrain from using any such materials or copies thereof. NYCERS’ materials consist of but are not limited to devices, records, data, notes, reports, proposals, business cards, letters, specifications, drawings, equipment, and other materials. Respondent shall agree to erase or destroy any NYCERS materials and portions thereof contained in all types of computer memory, and so warrant in writing to NYCERS within thirty (30) days of termination.

H.21. Permissible Work Locations The selected Respondent shall be available to meet, conference, or video chat with NYCERS during all business hours regardless of the selected Respondent’s location.



For all US-based resources, the selected Respondent shall take all necessary and appropriate steps to determine that all personnel are legally eligible to work in the United States, and that such eligibility is maintained at all times during which any such personnel is performing pursuant to the proposed solution.

Any work that NYCERS agrees may be performed off-site or off-shore, including cloud-based services, NYCERS may, in its discretion, at any time, during ordinary business hours, inspect the site or site(s) of the Contractor or any subcontractor engaged in contract performance. The Contractor shall guarantee NYCERS appropriate access to the site to perform such inspections.

Contractor must satisfy and comply with all NYCERS security requirements and may be subject to additional confidentiality requirements applicable to any such off-site and off-shore resources that NYCERS agrees to. In no event will work performed by resources off-site and/or off-shore be priced based upon an hourly rate greater than the rates applicable to on-site resources.

H.22. Firm Fixed Price for Complete Delivery of Scope and Requirements Cost for all services shall be a firm fixed price for the complete delivery of all scope and requirements regardless of the time and effort required by the selected Respondent and any subcontractors, including in cases where time and effort exceeds the selected Respondents original plans and estimates. Firm fixed price shall under no circumstances be interpreted as anything else. For instance, if an agile methodology is being proposed, a firm fixed price shall not mean a firm fixed price for a set number of development “sprints” regardless of the scope completed and/or requirements met at the end of the planned set of sprints. Or, for instance, if any given requirement is found by the selected Respondent to require more time and effort than originally planned, such as it was anticipated by the Respondent to be fulfilled “out-of-the-box” and then subsequently it is determined by NYCERS, the Respondent, or both, that the requirement can only be fulfilled through customization, the Respondent shall not be entitled to additional compensation (direct or indirect) or special considerations. Respondent agrees their proposed firm fixed price is reasonable and realistic and is not in any way a “minimum” or “starting” price. Respondent will not be entitled to additional compensation (direct or indirect or special considerations due to underestimating or underpricing the firm fixed price. Furthermore, by submitting a proposal, Respondent certifies it has requested clarification from NYCERS on all elements of the RFP during the written questions period (see Section H.5) necessary to furnish a compliant firm fixed price as described herein and have also updated their proposed firm fixed price and entire proposal submission (e.g., technical, cost) accordingly based on any NYCERS’ responses.

H.23. Required Deliverables Appendix 1 lists the deliverables required to be furnished by the vendor to NYCERS throughout the project.

H.24. Deliverable Expectations Document For each deliverable, the vendor will be required to provide a deliverable expectation document (DED) on a format NYCERS prescribes (see Appendix 2). Work performed on deliverables without a NYCERS-accepted DED is at the Contractor’s risk.



H.25. Final Deliverable Review A deliverable may be submitted for final review when the Contractor, in good faith, believes all requirements for that deliverable have been met, after all NYCERS feedback on any required draft deliverables has been addressed (see Appendix 2), and if in alignment with agreed upon timings. For deliverables NYCERS solely determines are eligible for final deliverable review, NYCERS shall have no less than fifteen (15) full business days to either accept the deliverable or provide written deliverable feedback to the Contractor, which the vendor shall be required to address completely and comprehensively prior to resubmission.

H.26. Payments Progress payments against the total cost of the phase or release will only be made when the vendor achieves a set of NYCERS-prescribed payment milestones as described in this section. Progress payments will not be made on a per deliverable basis. Any amounts retained will be released upon NYCERS full acceptance of all work and deliverables at the completion of the entire project. Contractor is responsible for preparing and furnishing invoices to NYCERS prior to any payments being made and such payments will be made consistent with NYCERS 30 days terms. All payments must be received from NYCERS via an electronic fund transfer (EFT).

Project Assessment Phase Only one payment will be made for the Project Assessment Phase (i.e., Phase 0). The Project Assessment Phase will be payable upon NYCERS full acceptance that the Phase has been completed successfully and NYCERS’ formal acceptance of all Project Assessment Phase deliverables identified in Appendix 1.

NYCERS will pay no more than $350,000 for the Project Assessment Phase. Any costs above this amount shall be the sole responsibility of the vendor and at no point will be payable. A 10% retainage shall apply to the Project Assessment Phase.

Phased Solution Releases Each release will have the following non-negotiable payment milestones. A payment milestone will be considered achieved if all associated deliverables and realized benefits as assigned to the milestone, as well as any updates to previously accepted deliverables to ensure they are current, are fully accepted by NYCERS (see Appendix 1). In addition, the Contractor must have successfully achieved all prior payment milestones leading up to the current payment milestone.

Payment Milestone % of Release Cost Paid

A 10%

B 10%

C 25%

D 45%

Retainage 10%



SLA Retainage An additional retainage of 0.5% for each SLA missed (see Appendix 4) for any reporting interval from the last payment milestone date to the current payment milestone date shall be withheld from the total payable amount of the current payment milestone (applies to Project Assessment Phase and all Release phases). Any SLA retainage withheld during the project will be released with other retainage. NYCERS will solely determine if an SLA has been missed and there shall be no limiting circumstances attached to SLA retainage. The selected Respondent is solely and exclusively responsible for achievement of SLAs and therefore the avoidance of SLA retainage. NYCERS shall in no way be held accountable for the selected Respondent’s inability to meet SLAs, and therefore the application of SLA retainage as a result, under any circumstances.

H.27. Off-Shoring Respondents are permitted to propose use of individuals working off-shore to the extent such proposal conforms with all parameters, terms and conditions set forth regarding such staffing in this RFP. No more than one-third (33.333%) of a Respondent’s project resources, as listed in the Respondent’s Cost Proposal (see Section F.2) may filled by individuals working off-shore. Respondents are advised that NYCERS at its sole discretion will decide whether off-shore resources will be authorized.

H.28. Vendor’s Acknowledgment of NYS Freedom of Information Law Vendor must complete and return the Vendor’s Acknowledgment of NYS Freedom of Information Law (Template G). Vendor shall give specific attention to the identification of those portions of their proposals the Vendor deems to be confidential, proprietary information or trade secrets and provide any justification why such materials, upon request, shall not be disclosed by NYCERS. Such information must be easily separable from the non-confidential section of the proposal.

H.29. RFP Inconsistencies It is the sole responsibility of Respondents to review all RFP materials in detail and identify any perceived, potential and/or actual inconsistencies within the RFP and to submit sufficiently detailed written questions to NYCERS during the designated written Respondent questions period requesting clarification. If inconsistencies are discovered later in the procurement process or after contract award, NYCERS will solely determine which version of the inconsistent element prevails. Respondents will be then required to follow NYCERS determination without any reduction in delivery (e.g., reduced scope, increased timeline) and without any increase in price to NYCERS.

H.30. City of New York Prompt Payment Policy Vendors are hereby notified of the City of New York’s Prompt Payment Policy:

(a) The prompt payment provisions of PPB Rule § 4-06 are applicable to payments made under this Agreement. The provisions generally require the payment to the Contractor of interest on payments made after the required payment date, as set forth in the PPB Rules.

(b) The Contractor shall submit a proper invoice to receive payment, except where the Agreement provides that the Contractor will be paid at predetermined intervals without having to submit an invoice for each scheduled payment.



H.31. Iran Divestment Act Compliance Rider for New York City Contractors The Iran Divestment Act of 2012, effective as of April 12, 2012, is codified at State Finance Law (“SFL”) §165-a and General Municipal Law (“GML”) §103-g (see Appendix 5). The Iran Divestment Act, with certain exceptions, prohibits municipalities, including the City, from entering into contracts with persons engaged in investment activities in the energy sector of Iran. Pursuant to the terms set forth in SFL §165-a and GML §103-g, a person engages in investment activities in the energy sector of Iran if:

(a) the person provides goods or services of twenty million dollars or more in the energy sector of Iran, including a person that provides oil or liquefied natural gas tankers, or products used to construct or maintain pipelines used to transport oil or liquefied natural gas, for the energy sector of Iran; or

(b) The person is a financial institution that extends twenty million dollars or more in credit to another person, for forty-five days or more, if that person will use the credit to provide goods or services in the energy sector in Iran and is identified on a list created pursuant to paragraph (b) of subdivision three of Section 165-a of the State Finance Law and maintained by the Commissioner of the Office of General Services.

A bid or proposal shall not be considered for award nor shall any award be made where the bidder or proposer fails to submit a signed and verified bidder’s certification (Template I).

Each bidder or proposer must certify that it is not on the list of entities engaged in investment activities in Iran created pursuant to paragraph (b) of subdivision 3 of Section 165-a of the State Finance Law (Template I). In any case where the bidder or proposer cannot certify that they are not on such list, the bidder or proposer shall so state and shall furnish with the bid or proposal a signed statement which sets forth in detail the reasons why such statement cannot be made. The City of New York may award a bid to a bidder who cannot make the certification on a case by case basis if:

(1) The investment activities in Iran were made before the effective date of this section (i.e., April 12, 2012), the investment activities in Iran have not been expanded or renewed after the effective date of this section and the person has adopted, publicized and is implementing a formal plan to cease the investment activities in Iran and to refrain from engaging in any new investments in Iran: or

H.32. (2) The City makes a determination that the goods or services are necessary for the City to perform its functions and that, absent such an exemption, the City would be unable to obtain the goods or services for which the contract is offered. Such determination shall be made in writing and shall be a public document.Doing Business Data Form Notice

1. Notice

Pursuant to Local Law 34 of 2007, amending the City's Campaign Finance Law, the City is required to establish a computerized database containing the names of any "person" that has "business dealings with the city" as such terms are defined in the Local Law. In order for the City to obtain necessary information to establish the required database, vendors responding to this solicitation are required to complete the attached Doing Business Data Form (“Data Form”) and return it with this proposal. (If the responding vendor is a proposed joint venture, the entities that comprise the proposed joint venture must each



complete a Data Form.) If the City determines that a vendor has failed to submit a Data Form or has submitted a Data Form that is not complete, the vendor will be notified by NYCERS and will be given four (4) calendar days from receipt of notification to cure the specified deficiencies and return a complete Data Form to NYCERS. Failure to do so will result in a determination that the proposal is nonresponsive. Receipt of notification is defined as the day notice is e-mailed or faxed (if the vendor has provided an e-mail address or fax number), or no later than five (5) days from the date of mailing or upon delivery, if delivered.

2. The Data Form

If you have already completed a Data Form (Template J), please submit it as a “No Change,” which you will find under Section 1. Entity Information. You only need to provide your Entity Name and EIN Number on the first page of the form and skip to the bottom of the 4th page and complete the certification box. If any of your information has changed you will need to complete the form as a “Change” under Section 1. Entity Information and update the sections that have changed. If you are unsure whether you have already completed the Data Form or when you last completed the Form, please call the Mayor’s Office of Contracts at (212) 788-8104 or you may send an email with your inquiry to [email protected].

H.33. “Appendix A” General Provisions Governing Contracts for Consultants, Professional, Technical, Human and Client Services – Notice to Proposers

The City of New York has issued a new Appendix A, General Provisions Governing Contracts for Consultants, Professional, Technical, Human And Client Services (see Appendix 7).

mailto:[email protected]



I. Evaluation and Contract Award This section contains information about how Respondents will be evaluated and contracts, if any, will be awarded. NYCERS is not obligated to award any contract as a result of this procurement. NYCERS also reserves the right to award contracts to multiple Respondents, at its sole discretion, as a result of this procurement.

I.1. Evaluation Procedures All proposals accepted by NYCERS will be reviewed to determine whether they are responsive or non-responsive to the requisites of the RFP. Proposals that are determined by NYCERS to be non-responsive will be rejected and not considered further. NYCERS’ Evaluation Committee will evaluate and score all remaining proposals as described in this section. Discussions may be conducted with Respondents submitting acceptable proposals and the panel may request a Best and Final Offer (see Section G). NYCERS may choose, at its sole discretion, to only engage in such activities with some Respondents but not others (e.g., Respondents with higher initial evaluation scores) and NYCERS reserves the right to adjust its initial evaluation accordingly based on the result of such activities throughout the procurement process (see Section G). However, NYCERS also reserves the right to award the contract on the basis of the initial proposals received, without discussions. Therefore, each Respondent’s initial proposal should contain its best terms.

I.2. Evaluation Criteria To be eligible for evaluation, proposals must be deemed responsive by NYCERS, include all components, and meet all proposal requirements. Scoring will be done on a weighting system against evaluation criteria as specified in Table 13. Exceptional proposals with regard to each criteria area shall receive a high amount of points for that criteria. Unacceptable or substandard proposals with regard to each criteria will receive fewer or no points for that criteria. NYCERS reserves the right to reduce a Respondent’s total score by any amount of points, without limitation, as a result of any RFP exceptions included in Respondent proposals that NYCERS deems, at its sole discretion, to be unfavorable to NYCERS.

Table 13. Evaluation Criteria

Criteria Weighting Organization Capability and Relevant Experience (quality, track record, and financials)

20%

Proposed Services (quality, approach, and methodology) 35% Proposed Project Staffing and Resource Management (quality of both staffing plan and individual proposed staff members)

25%

Cost 20% Total: 100%

I.3. Basis of Award Any contracts awarded will be on the basis of the proposals determined to be the most advantageous to NYCERS, taking into consideration the price and such other factors or criteria that are set forth in this RFP.



NYCERS reserves the right to evaluate and/or reject all bids, in whole or in part, and to waive or modify technicalities, irregularities, and omissions, or solicit new proposals if, in NYCERS’ judgment, the best interest of NYCERS will be served. Following bidding selection and prior to signing a contract, NYCERS reserves the right to further negotiate cost or other specifics. In the event the vendor is unable to fulfill their agreement with NYCERS, in part or in whole, NYCERS reserves the right, at its sole discretion, to evaluate a replacement from any other Respondents or seek the services from the Respondent with the next highest Final Composite Score.

Respondents are hereby notified that the prices stated in their proposals are irrevocable until the contract is awarded, unless the proposal is withdrawn. Offers may be withdrawn in writing to NYCERS no earlier than 90 days from opening and in advance of the award. NYCERS reserves the right to request a cost breakdown of the Respondent’s proposed price.

Documents

Request for Proposal for Software Quality Assurance and