1

Request for Solutions Advanced Distributed Learning (ADL) Initiative

cmi5 Player and Test Suite (PTS) Prototype Project 11 May 2020

1. Purpose and Authority

This Request for Solutions (RFS) is seeking vendors for an Other Transaction Authority (OTA) agreement, for the Advanced Distributed Learning (ADL) Initiative, cmi5 Player and Test Suite (PTS) Project, in accordance with the authority of 10 USC §2371b. The Government will evaluate the solutions with the intent to competitively award one Other Transaction (OT) Agreement for a prototype project through the Training and Readiness Accelerator (TReX). This effort will be an unclassified prototype project.

2. Summary and Background

2.1 Overview

Developed nearly two decades ago, the Sharable Content Object Reference Model (SCORM) is a set of interoperability standards for packaging and delivering online courses via web-browsers and Learning Management Systems (LMSs). However, SCORM is not extensible enough to support the myriad of technologies used in modern training and education environments, and SCORM does not provide sufficient guidance for capturing robust, interoperable learner performance data. As a result, a fundamentally new software capability is needed. This capability cannot simply be an improvement of SCORM, which was developed in a prior computing era before smartphones and cloud computing. An innovative and fundamentally new distributed learning capability is required. The cmi5 specification provides a better (more flexible, adaptable, and robust) alternative to current SCORM standards. The cmi5 specification is a set of rules that define how online courses are imported, launched, and tracked using an LMS along with the Experience Application Programming Interface (xAPI) and an xAPI Learning Record Store (LRS). The cmi5 specification contains and defines specific interoperability rules for content launch, authentication, session management, reporting, and the course structure. From an advancement of technology perspective, the cmi5 specification is one of many interdependent components required to enable the future learning ecosystem.

2

2.2 Program Objectives

The goal of this project is to help the DoD training and education community move away from legacy e-learning systems (i.e., SCORM-based and locked into browser-based LMS delivery) and instead move toward an integrated enterprise-wide learning ecosystem that (a) allows flexible delivery (e.g., content from outside an LMS or from different platforms) and (b) supports robust learning analytics. As envisioned, the cmi5 PTS prototype project will prototype a freely available, open source cmi5 content player prototype that can be implemented by DoD stakeholders or adopted for use by e-learning technology vendors as they start to support this modern specification. The cmi5 player will be able to import, register, authorize, launch, and track any cmi5-based learning content while also communicating the learner performance to an LRS. The CTS will be used by vendors and government offices to validate that cmi5 software applications as well as content packages adhere to the specification. Government offices will use this test suite to support the acquisition of cmi5 systems and content. The cmi5 PTS prototype project seeks to acquire innovative cmi5 prototype capabilities. The prototype will consist of two interrelated parts. First, the awardee of this project will design, develop, test, and transition a free and open source cmi5 Player that serves as a reference implementation for Department of Defense (DoD) stakeholders. Second, the awardee of this project will deliver a cmi5 Conformance Test Suite (CTS) that validates cmi5 learning systems (e.g., LMSs) and content packages against the latest version of the cmi5 specification (section 11). Key outcomes of the cmi5 PTS prototype project objectives include:

• Enable a recapitalization strategy for migrating existing instructional assets away from SCORM without gaps in mission performance while providing DoD stakeholders a more granular data tracking capability than SCORM, enabling deeper insights into learner progress and improved instructional feedback.

• cmi5 Content Examples and Authoring guide – Convert and test SCORM-based e-learning content into the cmi5 format. To aid DoD organizations, clear guidance and resources are required such as cmi5 conversion templates, example converted and tested courses, and an authoring guide to share best practices.

• Support DoDI 1322.26 and the adoption of xAPI across the DoD

enterprise while reducing licensing costs by providing an accredited cmi5 player to the community. Through the cmi5 PTS effort, it is expected that a prototype capability will be deployed, tested, validated, and

3

transitioned. Additional information about DoDI 1322.26 can be found here: https://adlnet.gov/policy/dodi-xapi/

• Deliver an extensible, open source cmi5 player prototype that serves as a

foundation for traditional Distributed Learning (DL) courses and content while also delivering a resource to the Training and Education community that can be built upon to add additional features and capabilities as needed by other DoD stakeholders. The cmi5 player will provide a basic capability to play cmi5-enabled courses required in order to provide easy access to a cmi5 player without other bundled software applications or third-party license fees. This basic player will provide a cost-effective capability to support software development and content migration efforts, and it will provide a fundamental capability for delivering cmi5 courses across the DoD.

• Deliver a prototype cmi5 Conformance Test Suite to validate cmi5 content packages to ensure adherence to the specification and to support conformance testing as part of the DoD content acquisition process. DoD requires some way to verify that cmi5-based software applications and cmi5-courseware is fully compliant to the cmi5 specification. A cmi5 CTS will enable DoD-wide testing and certification for cmi5-conformant applications and courses. The cmi5 CTS is expected to have broad applicability. For instance, all acquired cmi5-based distributed learning content should pass the conformance test prior to acceptance by the Government. A similar process was used for SCORM course acquisition; however, more advanced technological approaches are anticipated for the cmi5 CTS.

• Documentation – Finally, as for any software project, high-quality documentation is required for both the cmi5 Player and the cmi5 CTS. This should include an analysis of alternatives for the design of the two applications and a transition plan that defines the roadmap (e.g., requirements, cybersecurity, and compliance) for eventually implementing the applications on DoD networks.

The cmi5 PTS prototype project is a critical step in achieving these desired outcomes.

3. General Information

3.1. Vendors interested in responding to this RFS must be members of the Training and Readiness Accelerator (TReX). Information about membership can be found at the following webpage: https://nstxl.org/membership/

3.2. The cost of preparing and submitting a response is not considered an allowable

direct charge to any Government contract or agreement.

4

3.3. An individual vendor may not submit more than one comprehensive response to this RFS as a Prime. A vendor may participate as a subcontractor to multiple responses.

3.4. Non-compliance with the submission instructions provided herein may preclude

the vendor from being considered for award. 3.5. All Government participants and advisors in the evaluation process will be

required to sign non-disclosure agreements (NDAs).

4. Government Furnished Information (GFI)

4.1. Vendors are encouraged to utilize Links 1 – 9 referenced in Section 11, during Solution preparation. No other GFI is provided at this time.

4.2. Vendors must provide a list of all GFI that the vendor believes is critical to support Solution preparation.

5. Solution Paper Responses

5.0 Solution Paper responses shall include a separate General volume and a combined Technical and Price volume. No pricing detail shall be provided in any volume other than the Technical and Price Volume and shall be included in the Pricing section. Responses shall be submitted in an editable/executable (not scanned) Word/Adobe PDF format and limited to no more than 25 standard size (8 ½” X 11”) pages for the total volume count (see table below), using standard 12-point Arial font. Charts or figures are not bound by the 12-point font requirement but shall be clearly legible. Page size of 11”X17” are allowed for charts or figures only and each page will be counted towards the 20* page limit (see Table 1). If the solution exceeds the page limitation, the Government may choose not to read any information exceeding the 25* page limit and the information may not be included in the evaluation the solution. The Cover Pages, OCI & Mitigation Plan, and Data Rights Assertion are not included in the page total. All PDF’s will be editable (not locked). Table 1Volume Format Page Limit General Volume Word/PDF 5 Technical and Pricing Volume Word/PDF/Excel 20

Each volume shall include the following:

General Volume • Cover Page • Nontraditional status • Foreign Owned, Controlled or Influenced (FOCI) status

5

• Organizational Conflicts of Interest and Mitigation Plans Technical and Pricing Volume

• Cover Page • Sub-Vendor List • Vendor Experience • Project Management • Solution Paper • Government Desired Rights in Technical Data and Computer Software • Anticipated Delivery Schedule • Pricing Breakdown

5.1. General Volume

5.1.1. Cover Page

The cover page shall include the vendor’s name, Commercial and Government Entity (CAGE) Code (if available), NAICS Code, Business Size, address, primary point of contact, and status of U.S. ownership. NAICS code for this effort is 541512.

5.1.2. Nontraditional Status

The vendor shall provide its nontraditional (see paragraph 5.1.2.1. for definition) business status or its ability to meet the eligibility requirements of 10 U.S.C. §2371b. The vendor shall check one of the following boxes – with appropriate justification if needed.

There is at least one nontraditional defense contractor or nonprofit

research institution participation to a significant extent in the project. All significant participants in the transaction other than the Federal

Government are small businesses or nontraditional defense contractors. At least one third of the total cost of the project is to be provided by

sources other than the Federal Government.

If the vendor is not a nontraditional defense contractor (NDC) additional information is needed. Vendor shall provide the name and CAGE code information for the NDC. Additionally, the vendor shall provide what portion of the work the NDC is performing and an explanation of how the prototype would not succeed based on the portion of work performed by the NDC.

5.1.2.1. Definition Nontraditional – an entity that is not currently performing and has not

performed, for at least one-year period preceding the solicitation of sources by the Department of Defense (DoD) for the procurement or transaction, any contract or subcontract for the DoD that is subject to full coverage under the

6

cost accounting standards prescribed pursuant to 41 U.S.C §1502 and the regulations implementing such section.

5.1.3. Foreign Ownership, Control or Interest (FOCI) Status

In accordance with RFS Attachment 3, Security Process for Vendor Vetting, the General Volume must include certification that the vendor (and subcontractor(s)) are not Foreign Owned or under USA FOCI status (and are not in merger or purchasing discussions for a Foreign company or USA FOCI Company). Should a prospective vendor be unable to so certify, they will be ineligible for award unless the mitigating circumstances in Attachment 3 Security Process for Vendor Vetting are met. In such a case, these mitigating circumstances shall be detailed in an appendix to the General Volume.

5.1.4. Organizational Conflicts of Interest and Mitigation Plan

Vendors will submit an Organizational Conflict of Interest (OCI) Mitigation Plan via an appendix to its General Volume. In the event there are no real or perceived OCIs, simply state so and annotate what actions would be taken in the event that one is realized.

5.2. Technical and Pricing Volume

5.2.1. Cover Page

The cover page shall include the vendor’s name, CAGE code (if available), NAICS Code, Business Size, address, primary point of contact, and status of U.S. ownership. NAICS code for this effort is 541512.

5.2.2. Sub-Vendor List

Vendor shall provide a list of all sub-vendors involved and their role within the performance of your submission as an appendix to Technical Volume. The list shall include FOCI status and OCI.

5.2.3. Vendor Experience Vendor shall describe their company’s recent and relevant previous experiences developing e-learning systems similar to the required work of this prototype project. Projects worked in the last three years are considered recent.

5.2.4. Project Management

Vendor shall describe their company’s methodologies, organizational structure, quality assurance processes, and staffing they intend to use to manage this prototype project.

5.2.5. Solution Paper

7

Solution Paper responses shall include the vendor’s proposed technical solution describing the approach, feasibility and technical risks and mitigation solutions identified in fulfilling the Project Technical Objectives and associated deliverables identified below. The approach shall clearly address planned documentation deliverables (including format and content) and any planned demonstrations, design reviews (including product line quality factors such as agility and reuse), feasibility of implementation, total project risk, and management reviews.

5.2.6 Project Technical Objectives

cmi5 Working Group Collaborate with the cmi5 Working Group. Attend weekly meetings and share progress with the group, collect and document feedback, and refine deliverables in accordance with that feedback. Additional information is located here: https://aicc.github.io/CMI-5_Spec_Current cmi5 Player and Test Suite Design Collect and document software requirements. Comprehensively review the cmi5 specification and best practices as documented on the cmi5 GitHub site (https://aicc.github.io/CMI-5_Spec_Current/) as well as the xAPI specification/IEEE 9274.1.1 standard (https://github.com/adlnet/xapi-profiles). Based on review and analysis, create system requirements for a web-based cmi5 player and a separate cmi5 CTS. Documentation must be in the form of a requirements traceability and verification matrix or equivalent as agreed to by the Government.

The cmi5 content player will function as an authenticated, web-based system that connects to an LRS, can import content from a single list of files, and sequences that content dynamically based on cmi5.

The cmi5 CTS is an authenticated, web-based system that validates both cmi5 courses/content and cmi5 software applications The ADL Initiative LRS Test Suite (lrstest.adlnet.gov) provides a model for the desired functionality. System Architecture Report. The System Architecture Report (SAR) describes the specification and deployment of hardware and software, interfaces and port/protocol usage, and other architecturally relevant information. The architecture shall include a description of the concept of execution, which may be developed in concert with the User Interface/ User Experience Design Documents. A single System Architecture Report will be created for the cmi5 Player and another, separate report will be created for the cmi5 CTS. UI/UX Designs. The Graphical User Interface (GUI) shall incorporate principles of user-centered design (UCD) for user experience (UX) and user interfaces (UIs).UI/UX storyboards, design wireframes, and documentation will be made available during technical interchange meetings with the ADL Initiative in time

8

to support review and comment prior to final design of the prototype cmi5 Player and CTS. This includes the multiple roles of learner, instructor, cmi5 conformance test user, and administrator. Document Requirements for cmi5 software test packages. The purpose of a CTS is to validate that all aspects of cmi5 (and by inheritance, xAPI) are followed in accordance with the technical requirements. These technical requirements take on the form of tests, which are then “packaged” in software. The contractor must use the list of tests developed by the cmi5 Working Group as a basis but may modify them as necessary. All modifications must be done in collaboration with the cmi5 Working Group. Documentation must be in the form of a Requirements Traceability and Verification Matrix (RTVM) or equivalent as agreed to by ADL Initiative. Test and Evaluation Plan. Each requirement listed in the RTVM must be succinctly defined, measurable, and testable. The Test and Evaluation (T&E) Plan will outline the process used to test each requirement. This will be used to support Government Acceptance Testing of the prototype software. cmi5 Player and Test Suite Development Create cmi5 content player prototype. Software must be built in accordance with all documented requirements. Software code must be accessible in GitHub, updated at least monthly (the performer is not obligated to push all updates to the software immediately to GitHub). This prototype shall be designed to integrate with the Total Learning Architecture (TLA)1 Reference Implementation located within the TLA Sandbox,2 a cloud-based development environment operated by the ADL Initiative. It must also be able to run in a cybersecurity-accredited DoD environment as an independent deployment. This capability should be developed to best align with DoD’s Risk Management Framework (RMF) and should be targeted to run in an IL2 environment. An ATO is not within the scope of this effort but will be pursued after further testing and evaluation of the prototype. Create cmi5 Conformance Test Suite prototype. Software and test packages must be built in accordance with all documented requirements. Software code must be accessible in GitHub, updated at least monthly. Software must follow cybersecurity procedures as documented above. This prototype shall be designed to operate on the TLA Sandbox cloud environment operated by the ADL Initiative. It must also be able to run in a cybersecurity-accredited DoD environment as an independent deployment. Developer Verification Test. Upon completion of all prototype development activities, a verification test shall be performed on the cmi5 Player and CTS

1 Information about the TLA is located at https://adlnet.gov/projects/tla/ 2 TLA Sandbox summary: https://adlnet.gov/news/2020/01/20/ADL-Initiative-established-a-TLA-Sandbox-project/

9

prototypes using Government-approved testing procedures. Testing procedures will be outlined in the T&E Plan. The results will be described in a cmi5 PTS System Test Report (STR). Integrate with TLA Reference Implementation. The cmi5 Player prototype will be integrated into the ADL Initiative’s TLA Reference Implementation and hosted in the TLA sandbox to support additional testing and evaluation by the Government. The cmi5 CTS prototype will also be installed to run within the TLA Sandbox for further Government testing and evaluation. Government Acceptance Test. Thoroughly test the cmi5 Player and CTS prototypes to provide a functional testing report with the results. This testing should include software benchmarking as well as usability studies that are outlined in the T&E Plan. Tests will continue until all requirements have been satisfactorily verified. All identified anomalies shall be corrected before final acceptance by the Government. Capability Demonstrations. A walkthrough of all features and functionality of the cmi5 Player and Conformance Test will occur with the ADL Initiative stakeholders, cmi5 Working Group members, and other distinguished guests. Demonstrations shall take on the form of two 60-minute webinars that will be recorded. The ADL Initiative will provide the webinar delivery platform and administer the webinar. cmi5 Player and Test Suite Documentation cmi5 Player User Documentation. User documentation must be provided for learners, instructors, and administrators. This documentation should include “how to” instructions for how to log-in, register, navigate, and troubleshoot the cmi5 Player. cmi5 Conformance Test Suite User Documentation. User documentation must be provided for administrators and developers wishing to test their products using the CTS. Documentation will be provided to support learning technology vendors seeking to have their products certified for delivering cmi5 conformant courses and also for course developers seeking to have their courses certified as cmi5 conformant. System Installation and Administration Instructions. Detailed instructions shall be provided to define the steps and activities required to install and configure either prototype capability. These instructions shall contain an overview of the system, a description of the major tasks involved in the setup, configuration, and deployment of overall resources that are needed to get each capability operational. cmi5 Exemplars Create five (5) cmi5 course templates. Create cmi5 course templates that span

10

the range of complexity from the simple conversion of a legacy SCORM course that is 100% browser-based to blended courses that utilize a wide range of different instructional activities. Templates should include all instructional practices found in cmi5 (e.g., all verbs from cmi5 must be used). Best practices guide. This documentation will function as a “how-to” guide for legacy content migration. This documentation must include specific processes that describe how to apply cmi5/xAPI specifications. General Objectives (generic requirements applicable to all ADL Initiative projects) ADL Initiative specification maintenance: As applicable, the performer shall provide feedback, recommended edits, and/or suggested additions to the cmi5 standard and other fungible references for DoD Instruction 1322.26. For instance, this may apply if the performer identifies an error or flaw on a specification. Personally Identifiable Information. The performer shall comply with Federal regulations, including DoDI 8500.01 (“Cybersecurity”) and DoDI 8510.01 (“Risk Management Framework”), on the protection of Personally Identifiable Information (PII), as applicable. Human Research Protections. The performer shall adhere to human-subjects research regulations including guidance provided by ADL Initiative at the kick-off meeting (refer also to the Federal Policy for the Protection of Human Subjects, or “Common Rule,” published by the U.S. Department of Health and Human Services). Project profile: Performer shall develop and deliver a “project profile” using the Government template provided at the kick-off meeting. This includes an optional proposed logo, one-page executive summary, and short slideshow brief that includes the executive summary and any graphics. The profile will be updated as required with agreement from the performer and ADL Initiative. As the cmi5 capabilities are deployed across the DoD, they will need to meet Federal requirements for 508 compliance. Section 508 Amendment to the Rehabilitation Act of 1973, as amended in 1998 requires Federal agencies to make their electronic and information technology accessible to people with disabilities (see https://section508.gov).

Summary of Deliverables and Exit Criteria

The below table depicts direct deliverables associated with section 5.2.6 Project Technical Objectives. Deliverables will be evaluated based upon ADL technical experience, provided guidance, and collaboratively identified millstones in relation to exit criteria and tasking. Deliverables that do not satisfy the intent will have the opportunity to be refined per collaboration and guidance from the Advanced Distributed Learning Initiative.

11

Table 2. Tasks and Deliverables Summary Task Name Deliverable Due Exit Criteria

cmi5 Working Group Participation

Meeting minutes Continuous cmi5 Meeting Attendance Documented

cmi5 PTS Design Software Requirements

RTVM 2 Months ACA3 and updated as required

Requirements meets all technical objectives outlined in Section 5.2.6 and meets draft requirements defined by the cmi5 Working Group

System Architecture Report

System Architecture Report

3 Months ACA Architecture shows clear and comprehensive practices for data-intensive software systems.

UI/UX Design User Interface/ User Experience Design Documents

3 Months ACA Design documents reflect a well-designed user experience and pass usability acceptance testing as defined in the T&E Plan.

Test and Evaluation Plan

T&E Plan 3 Months ACA Test plan addresses all defined requirements from the SOW based on the technical objectives reference in

cmi5 Conformance Test Suite Packages Requirements

RTVM 8 Months ACA RTVM concurrence by cmi5 Working Group and acceptance by ADL Initiative

cmi5 PTS Dev – cmi5 Player

cmi5 content player software and documented code

10 Months ACA Software successfully passes Government Acceptance Test as described in Section 5.2.6, Test and Evaluation Plan

cmi5 PTS Dev – cmi5 Conformance Test Development

cmi5 conformances test suite software, software packages, and documented code

10 Months ACA Software successfully passes Government Acceptance Test as described in Section 5.2.6, Test and Evaluation Plan

Developer Verification Test

Developer Verification Test Report

11 Months ACA No major anomalies

3 ACA = After contract award

12

Table 2. Tasks and Deliverables Summary Task Name Deliverable Due Exit Criteria

Government Acceptance Test

GAT Test Report

11 Months ACA No major anomalies

Capability Demonstrations

Two – 60minute webinars using

12 Months ACA Completion of Webinars

Cmi5 Player User Documentation

Instructor, Learner, Administrator Guide

11 Months ACA Documentation is accepted by the ADL Initiative

cmi5 CTS Documentation

Developer and administrator guides

11 Months ACA Documentation is accepted by the ADL Initiative

cmi5 PTS Implementation Instructions

System Administrators Guide

11 Months ACA Documentation is accepted by the ADL Initiative

Cmi5 Course Examples

cmi5 courses examples (10)

11 Months ACA Each example must run in the cmi5 content player and pass the cmi5 Conformance Test

cmi5 Best Practices Guide

cmi5 conformance test suite end-user instructions and documentation

11 Months after Contract Award

Documentation is accepted by the ADL Initiative

5.2.7 Government Desired Rights in Technical Data and Computer Software

5.2.7.1 For the purposes of this RFS and final award document, the Government will use the data rights and computer software related terms defined in Attachment 2, Data Rights License Terms Definitions.

5.2.7.2 The performer shall provide the Government unlimited rights to works and data first produced, created or generated, and required to be delivered under this agreement. The Government will collaborate with the performer to determine which, if any, of these deliverables should be made open source. Open-source software developed under this project shall be released under an Apache 2.0 open-source license or similar licenses acceptable to the performer and Government. The performer shall deliver all algorithms with full use rights under a license approved by the Government. The performer shall not include any proprietary (non-open source) software or media in any deliverable without prior permission of the Government. The performer shall grant Unlimited Rights to release or disclose technical data outside the Government and authorize persons

13

to whom it is released or disclosed to use, modify, reproduce, release, perform, display, or disclose that data for U.S. Government purposes. 5.2.7.3 Commercial Computer Software If the Government has a need for rights not conveyed under the license customarily provided to the public, the Government must negotiate with the third party software vendor(s) to determine if there are acceptable terms for transferring such rights. The specific rights granted to the Government shall be enumerated in the license agreement or an addendum thereto. The Government shall only have the rights specified in the license for the commercial computer software and its related commercial computer software documentation for the software listed in the Data Rights Assertion table. The terms of any End User License Agreements apply only to the extent they are consistent with law and Attachment 4, Terms and Conditions and EULA. 5.2.7.4 The vendor shall describe the intellectual property rights being provided to the Government in terms of technical data and software, clearly outlining any rights restrictions. If the proposed solution includes commercial software, copies of any applicable End User License Agreements (EULAs) must be submitted with the response. It is the Government’s intent to plan for the concurrency, maintenance, and modification of the Suite using Government personnel and third party contractors. The EULA submissions have no page limit and do not count against the proposed technical solution page limitation. 5.2.7.5 The vendor shall make every effort to incorporate non-proprietary resources and tools when feasible and applicable to the effort. This includes, but is not limited to, software rights, data, source code, drawings, manuals, warranties, and integration efforts.

5.2.8 Anticipated Delivery Schedule

The vendor shall include the anticipated delivery dates with their solution that includes all ADL cmi5 capabilities and completion dates for all tasks as described in the RFS. The vendor is expected to use the Tasks and Deliverables Summary (Table 2 in section 5.2.6) in preparation of their proposed delivery schedule.

5.2.9 Pricing Breakdown

Vendors shall submit a firm fixed price amount for its solution, further divided into severable milestones. The Government is not dictating a specific price mechanism. However, proposed payments should be linked to clearly definable, detailed milestones. It should be clear, with sufficient detail, what is being delivered at each milestone. Each milestone price should reflect the anticipated value the Government will receive toward accomplishment of the

14

OTA goals and objectives at the time the milestone is completed. The price volume has no page number limitation. In addition, vendors shall describe their licensing/pricing model(s) for their described Prototype solutions.

It is important to note, the prototype project has a maximum ceiling budget of $898,568.

6. RFS Response Instructions

6.1. The Government intends to make a single OT award as a result of the RFS.

6.2. All questions related to this RFS shall be submitted via email to

initiatives@nstxl.org, with “ADL cmi5 Prototype Vendor Questions” in the subject line.

6.3. Questions must be submitted no later than 12:00 PM EDT on 18 May 2020.

Questions received after the deadline may not be answered. Questions shall not include proprietary data as the Government reserves the right to post submitted questions and answers, as necessary (and appropriate) to facilitate vendor solution responses.

6.3.1. The Government reserves the right to post submitted questions and answers,

as necessary (and appropriate) to facilitate vendor Solution Paper responses. Submitted questions will be posted without identifying company names.

6.4. Solution Responses shall be submitted no later than 12:00 PM EDT on 12 June

2020. Solution Responses shall be submitted electronically to initiatives@nstxl.org, with “ADL cmi5 Prototype” used in the subject line. Any submissions received after this time on this date may be rejected as late and not considered.

6.4.1. Vendors must clearly state assumptions made within their response. Vendors

are encouraged to challenge any Government assumptions or restrictive requirements in its individual solution and should articulate any major discrepancies between the RFS and Attachments, and its technical solution. Should a vendor’s solution require a change in policy and/or statue, the vendor shall outline within their technical volume, and describe why the change is needed to realize the benefit of the vendor’s prototype (and potential production).

7. Evaluation and Selection Process 7.1. The evaluation will consist of evaluation of the Solution Paper Response. If

further information is required after the completion of the evaluations of the

15

written Solution Paper Responses, the Government may request presentations and/or demonstrations or enter into communications with vendors. The Government will evaluate the degree to which the submission provides a thorough, flexible, and sound approach in response to the ability to fulfill the requirements.

Individual responses will be evaluated with consideration given to the overall technical merit and feasibility of the response and the total project risk with consideration aimed at the Project Technical Objectives referenced in Section 5 and the ability to satisfy and fulfill all the RFS requirements. The Project Technical Objectives, Vendor Experience, Project Management, Anticipated Delivery Schedule, Government Desired Rights in Technical Data and Computer Software and Pricing Breakdown will be considered as aspects of the entire response when weighing risk and reward. The Vendor’s focus is to describe its approach to the development of the prototype, as outlined in this RFS. The solution paper will be evaluated, and should provide specific emphasis, based upon the focus areas outlined in the below table. While these focus areas are of significant importance, responses will be considered as a whole. The Government evaluators will consider the following focus areas, in no specific order of importance.

Table 4. Focus Areas Focus Area Focus Area Description

1 Technical Merit and Feasibility 2 Demonstrated Experience 3 Project Management 4 Anticipated Delivery Schedule 5 Government Desired Rights in Technical

Data and Computer Software 6 Solution Price

7.1.1 Focus Area 1 – Technical Merit and Feasibility

The Government will evaluate the vendor’s technical approach, technical Merit and feasibility, as well as technical risk/mitigation solutions to meet the objectives of the cmi5 project identified in the Project Technical Objectives and associated deliverables. The Government will also evaluate the planned documentation deliverables (including format and content) and any planned demonstrations, design reviews (including product line quality factors such as agility and reuse), feasibility of implementation, total project risk, and management reviews.

7.1.2 Focus Area 2 – Demonstrated Experience

16

The Government will evaluate the vendor’s recent and relevant previous experiences developing e-learning systems similar to the required work of this prototype project. The Government will assess its level of confidence and risk based on the extent of demonstrated experience.

7.1.3 Focus Area 3 – Project Management

The Government will evaluate the vendor’s methodologies, organizational structure, quality assurance processes, and staffing intended to be used to manage this prototype project. The Offeror’s management approach will be evaluated to determine the extent to which the Offeror has developed a strategy for the effective and efficient management of contract activities, organizational structure, quality assurance processes, and staffing needed for this prototype.

7.1.4 Focus Area 4 – Anticipated Delivery Schedule

The Government will evaluate the Offeror’s proposed approach and understanding of the entire effort as demonstrated in the proposed delivery dates for all tasks as described in the RFS.

7.1.5 Focus Area 5 – Government Desired Rights in Technical Data and Computer Software

The Government will evaluate the vendor’s Attachment 1, Data Rights Assertions Tables, specifically their response to requested Government Desired Rights in Technical Data and Computer Software and evaluate for risk(s) impacts.

7.1.6 Focus Area 6 – Solution Price

The Government will evaluate the vendor’s overall price and how the proposed payments are linked to clearly definable, detailed milestones.

7.2. Selection Process

The Government anticipates awarding one OT prototype project, through TReX, to the vendor that proposes a solution that best satisfies the Government’s objectives.

7.2.1. The Government will review each vendor’s submittal against the criteria in

Section 7, with major consideration given in no specific order of importance to focus areas discussed above. Further, the Government will evaluate the degree to which the proposed solution provides an innovative, unique – yet realistic and sustainable - approach to meeting the cmi5 player and cmi5 test suite technical capabilities and objectives.

7.2.2. Assessment of risk is subjective. If the risk is obvious or the schedule seems

overly aggressive, the Government will consider that in the total risk assessment.

17

Vendors are responsible for identifying risks within their submissions, as well as providing specific mitigation solutions. If sufficient validation of the proposed information is not provided, the Government may reject the submission.

7.2.3. Unsupported assertions may be discounted by the evaluators.

The Government reserves the right to award to a vendor that does not meet all of the requirements but provides attributes or partial solutions of value.

7.2.4. In making the final decision it may become necessary to compare the proposals

of each vendor against the other, but the Government anticipates that its decision is more likely to be made based on each vendor’s submittal as evaluated against the criteria described above and a determination of which proposal(s) is/are determined to be the most advantageous to the Government.

8. Additional Information

8.1. Statement of Work

The proposed solution will be used to collaboratively negotiate a Statement of Work (SOW) for the cmi5 Player and cmi5 Test Suite prototypes after selection for award. The SOW will be incorporated into the OTA agreement as part of the award.

8.2. Export Controls Research findings and technology developments arising from the resulting proposed solution may constitute a significant enhancement to the national defense and to the economic vitality of the United States. As such, in the conduct of all work related to this effort, the recipient will comply strictly with the International Traffic in Arms Regulation (22 C.F.R. §§ 120-130), the National Industrial Security Program Operating Manual (DoD 5220.22-M) and the Department of Commerce Export Regulation (15 C.F.R. §§ 730-774).

8.3. Interaction and/or Disclosure

The Vendor should comply with foreign disclosure processes described in US Army Regulation (AR) 380‐10, Foreign Disclosure and Contacts with Foreign Representatives; Department of Defense Directive (DoDD) 5230.11, Disclosure of Classified Military Information to Foreign Governments and International Organizations; and DoDD 5230.20, Visits and Assignments of Foreign Nationals. All submissions will be unclassified. Submissions containing data that is not to be disclosed to the public for any purpose or used by the

18

Government except for evaluation purposes will include the following sentences on the cover page:

All submissions will be unclassified. Submissions containing data that is not to be disclosed to the public for any purpose or used by the Government except for evaluation purposes will include the following sentences on the cover page: “This submission includes data that will not be disclosed outside the Government, except to non-Government personnel for evaluation purposes, and will not be duplicated, used, or disclosed -- in whole or in part -- for any purpose other than to evaluate this submission. If, however, an agreement is awarded to this Company as a result of -- or in connection with – the submission of this data, the Government will have the right to duplicate, use, or disclose the data to the extent agreed upon by both parties in the resulting agreement. This restriction does not limit the Government's right to use information contained in this data if it is obtained from another source without restriction. The data subject to this restriction are contained in sheets [insert numbers or other identification of sheets]” Each restricted data sheet should be marked as follows: “Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this submission.”

Public Release or Dissemination of Information. There shall be no dissemination or publication, except within and between the Performer and any subcontractors, of information developed under this Agreement or contained in the reports to be furnished pursuant to this Agreement without prior written approval of the AOR/PM. All technical reports will be given proper review by appropriate authority to determine which Distribution Statement is to be applied prior to the initial distribution of these reports by the Performer.

9. Follow-On Production All deliverables, in accordance with Section 5, will be open source. Vendors will facilitate transfer of open source software to the ADL Initiative upon completion of the project. The ADL Initiative will continue to host prototypes as open source. Relevant technical reports will be published via the Defense Technical Information Center (DTIC). It is anticipated that potential follow-on production contracts may be awarded upon successful completion of the PTS project effort and may be awarded to the vendor without the use of competitive procedures. Pursuant to 10 U.S.C. 2371b(f), if a prototype pilot is successful, production OTs are authorized to offer a streamlined method for transitioning into follow-on production without competition. Successful completion will be defined as completion of project

19

technical objectives, tasks, submission of deliverables, and Government validation and acceptance of the prototype. The objectives, tasks, deliverables, validation and acceptance criteria are further defined in the negotiated SOW for this prototype project. For potential follow-on production efforts, cloud solutions are required, at a minimum, to meet FedRAMP Impact Level (IL) 2 (see https://www.fedramp.gov). The prototype cmi5 player in particular will need to be accredited in the future to operate within a minimum IL4 environment due to the collection and aggregation of learner data. For potential follow-on production efforts, the Government must be able to obtain an Assessment and Authorization (A&A) when interfacing with the vendor solution in accordance with DoD Instruction 8510.01, RMF for DoD IT. Other potential follow on work will be the integration of the prototype with the DoD’s Federated Identity, Credential, and Access Management (FICAM) policy to allow learner data to be aggregated across separate DoD networks. While the immediate focus for this project is on the functional testing and evaluation of the resulting prototypes, the overall goal of this work is to support DoD training and education modernization efforts, to including updates to the applicable policy: DoD Instruction 1322.26. As these prototype capabilities mature, the ADL Initiative in coordination with the Defense ADL Advisory Committee will update DoDI 1322.26 to recommend processes that specifically involve prototypes for use in acquisition and testing processes.4 Other follow-on activities may include the addition of adequate security controls to manage Controlled Unclassified Information (CUI), Personally Identifiable Information (PII), or other covered defense information that is processed, stored, or transmitted across the network. To provide adequate security, the performer must, at a minimum, implement National Institute of Standards and Technology (NIST) Special Publication (SP) 800-171, “Protecting Controlled Unclassified Information in Nonfederal Information Systems and Organizations.” Regardless of follow-on activities, the performer must report cyber incidents that affect a covered performer information system or the covered defense information residing therein. The performer must also submit malicious software discovered and isolated in connection with a reported cyber incident to the DoD Cyber Crime Center and if requested, submit media and additional information to support damage assessment. The performer must flow down these requirements to subcontractors for which subcontract performance will involve covered defense information.

4 See https://adlnet.gov/policy/dodi for more information and resources about DoDI 1322.26.

20

10. Attachments: Attachment 1, Data Rights Assertions Tables Attachment 2, Data Rights License Terms Definitions

Attachment 3, Security Process for Vendor Vetting Attachment 4, Terms and Conditions and EULA

11. Links: To support the cmi5 PTS RFS, the following links and/or documents will be provided and/or made available. All relevant cmi5 resources and documentation is publicly available (vendor vetting is not required):

1. Introduction to cmi5. This link provides background information on the cmi5

specification, how it works, what it enables, and benefits of using it. https://adlnet.gov/resources/cmi5-resources/

2. cmi5 Working Group. This link provides information on the cmi5 Working Group as well as information for participating in the group. https://github.com/AICC/CMI-5_Spec_Current/wiki

3. cmi5 Specification. This link provides the actual cmi5 specification and its technical documentation. A brief discussion on the history of cmi5 is also provided as well as additional links to find out more. https://aicc.github.io/CMI-5_Spec_Current/

4. cmi5 Best Practices. This link provides a list of best practices for the successful utilization of the cmi5 specification. https://aicc.github.io/CMI-5_Spec_Current/best_practices/

5. cmi5 Code Examples. This link provides a list of code samples that represent various scenarios that are possible when running content utilizing the cmi5 runtime definitions. https://aicc.github.io/CMI-5_Spec_Current/samples/

6. xAPI Specification. This is a link to the Github repository that contains the xAPI specification. In addition to the specification, there are markdown files containing fixes to some common issues. cmi5 uses xAPI, so the two specifications are tightly integrated. https://github.com/adlnet/xAPI-Spec

7. xAPI Technical Advisory Group. This link provides information on the TAG xAPI Working Group, which is open to the public, with the goal of assisting in the adoption of xAPI. Information may be found here on xAPI use cases or participating in the monthly calls. https://www.tagxapi.org/

8. xAPI Sample LRS. This link to the ADL Initiative reference implementation of a Learning Record Store (LRS). This LRS is not meant to function as a product-quality application. Rather, it provides an open-source barebones example. https://lrs.adlnet.gov/

21

9. SCORM Conformance Test Suite. This link provides the SCORM 2004 (3rd

Edition) Conformance Test Suite. The updated CTS allows organizations to test SCORM content within a more modern set of software. https://adlnet.gov/projects/scorm-2004-3rd-edition-conformance-test-suite/