English Language Proficiency Assessment for the 21st ... Field Test RFP_Questions... · English Language Proficiency Assessment for the 21st Century (ELPA21) Oregon Enhanced ... Number

English Language Proficiency Assessment for the 21st Century (ELPA21) Oregon Enhanced Assessment Grant

“THE CONTENTS OF THIS DOCUMENT WERE DEVELOPED UNDER A GRANT FROM THE U.S. DEPARTMENT OF EDUCATION. HOWEVER, THOSE CONTENTS DO NOT NECESSARILY REPRESENT THE POLICY OF THE U.S. DEPARTMENT OF EDUCATION AND YOU SHOULD NOT ASSUME ENDORSEMENT BY THE FEDERAL GOVERNMENT.” TO

COMPLY WITH PUBLIC LAW 112-74, “CONSOLIDATED APPROPRIATIONS ACT, 2012”, DIVISION F—DEPARTMENTS OF LABOR, HEALTH AND HUMAN SERVICES, EDUCATION, AND RELATED AGENCIES APPROPRIATIONS ACT, 2012, TITLE V—GENERAL PROVISIONS, SEC. 505, DECEMBER 23, 2011: 100% OF

THE ELPA21 PROJECT ARE FINANCED WITH FEDERAL MONEY; THE FEDERAL FUNDS FOR ELPA21 TOTAL $6,273,320; AND 0% OF THIS PROJECT IS FINANCED BY NON-GOVERNMENTAL SOURCES.

Vendor Questions Submitted Requesting Further Information or Clarification Regarding the ELPA21 Field Test RFP

# RFP Section RFP Page

Number

Related Text in RFP Question and Answer

1 General Question

What is the budget for this scope of work? One million dollars. ($1,000,000.00)

2 3.2 Bullet #6 7 Does “tracking” refer to being able to track student records across years?

Yes.

3 Overview/Sect ion 5 Functional and Technical Requirements for Field Test Development

14, 20 (p 14) These assessments will measure the students’ level of English proficiency in the four domains of reading, writing, speaking, and listening.

(p 20) Forms will be developed according to the specifications provided by the Council in conjunction with the item development vendor.

Are recommended example items of all item types publically available or available to the bidders?

No sample items are available at this time.

4 2 Major Tasks and Services

15 Are Large Print and Braille forms needed for the Screener?





As required in the RFP (pages 22 and 23), the contractor should develop one field test form in Braille and one field test form in large print for each grade/grade band.

5 2 Major Tasks and Services

15 Will the consortium member states recruit the participants in pilot and field-test studies?

The sampling and recruitment for the pilot study and field test will be the responsibility of the selected vendor, with the support of ELPA21 Assessment Design, Field Test, and Scaling (ADFTS) task management team (TMT) and the member states. The ADFTS TMT and/or member states will help in obtaining contact information and profile information (e.g., location and distribution of the ELL population), providing supportive communication, and troubleshooting when recruitment issues occur.

6 General What role, if any, is the contractor expected to play in the recruitment of schools for the field test and the possible limited pilot study?

Please see the response to #5.

7 2. Major Tasks and Services and 3. Deliverables

15, 17 Conduct range finding meetings to define scoring criteria for constructed- response items.

Table 1: Schedule of Major Deliverables - Assessment Research and Reporting

Are these meetings primarily volunteers from the Council with participation from the contractor’s Hand scoring Project Lead?

What are the expectations for travel/reimbursement of range finding participants?

How many participants?

How many days?

What of participation is expected from the vendor?





While educators and department staff from ELPA21 member states will participate in range finding, we expect the vendor to provide sufficient staff with ELL assessment expertise to facilitate the range finding meeting. The numbers of participants and days needed should be based on the assumption that 50% of the CR items will be scored on a 0-2 point scale, 30% on a 0-3 point scale, and 20% on a 0-4 point scale.

For bidding purposes, assume that 2 teachers and one state department staff member will participate per grade band, for a total of 18 external participants. The vendor will be responsible for travel reimbursement for all external participants and for providing substitute pay (average of $120/day) for teachers.

8 3. Deliverables 16, 17 Table 1: Schedule of Major Deliverables

What meetings on the schedule can the contractor expect to be piggybacked either before or after a CCSSO SCASS meeting?

Representatives from ELPA21 states meet face-to-face 2-3 times per year. Two meetings will typically occur in October and February on the Monday in which CCSSO’S SCASS meetings take place. If time permits, a vendor meeting may be added to the Thursday or Friday of those weeks.

9 Meetings 16, 18- 19

Where will in-person meetings (kickoff meeting, range finding, etc.) be held?

The location should be mutually agreed upon by the contractor and the Council.

10 Scoring 16, 30 (p 16) Computer-based Objective In reference to “computer-based objective scoring”, is ELPA21





Scoring—Utilize computer-based objective scoring methods to score appropriate constructed- response items.

(p 30) Computer-based objective scoring will be used for selected- response items, technology- enhanced items, and some constructed-response items which can produce a look-up list from student responses to the field test items or for which there is a finite, definable set of correct responses.

interested in the automated scoring of essays and short answer responses for the operational test?

This RFP does not include an operational test. The proposal should be based on a field test, and automated scoring of essays and short answer responses will not be used for this test.

11 3. Deliverables 17 Complete field test sample selections; Assist districts and schools in completing registration of field test samples

a) Will states conduct sample recruitment?

b) How will the contractor be expected to assist the states/ELPA21 to recruit sample participants?


12 3, Table 15, Field Test Forms

17, 22 Section 5 indicates that the test window will be several weeks in February and March of 2015. In Table 1, Assessment Research and Reporting section indicates that range finding will occur in March.

Please confirm that there will be enough field test responses available to prepare for range finding in March when testing is not complete until the latter part of March.

We expect that the contractor will have enough responses by early March for range finding. If the responses at some score points are not enough, the contractor should propose possible solutions to the Council for approval.

13 18 The last line in Table 1, Schedule of Major Deliverables, indicates





that the Offeror is to “submit open source system codes to Council” in June 2015. Please clarify.

The constructor should submit open-source codes to the Council at the end of the contract. However, the Council will consider a bidder that may meet interoperability criteria without strictly adhering to “open-source codes.” Please note that the Council must meet specific guidelines from US DOE (see below.) Should the proposer not adhere to “open-source codes,” then the proposer should explain how the company would meet the following guidelines. Any proposal that meets the “open-source codes” guidelines will be more highly evaluated than a proposal that does not.

84.368.A-1 (Federal Register/Vol. 77, No. 83/April 30, 2012/ Notices, page 25459) (f) Maximize the interoperability of any assessments and other assessment related instruments developed with funds from this competition across technology platforms and the ability for States to move their assessments from one technology platform to another by doing the following, as applicable, for any assessments developed with funds from this competition by—

(1) Developing all assessment items in accordance with an industry-recognized open-licensed interoperability standard that is approved by the Department during the grant period, without nonstandard extensions or additions; and

(2) Producing all student-level data in a manner consistent with an industry recognized open-licensed interoperability standard





that is approved by the Department during the grant period; (g) Unless otherwise protected by law or agreement as proprietary information, make any assessment content (i.e., assessments and assessment items) and other assessment-related instruments developed with funds from this competition freely available to states, technology platform providers, and others that request it for purposes of administering assessments, provided that those parties receiving assessment content comply with consortium or State requirements for test or item security.

14 Table 1, Schedule of Deliverables

18 Please clarify what open source system codes the Council expects to be delivered in June of 2015.


15 3. Deliverables 18 Table 1 in the RFP presents a Schedule of Major Deliverables. In the section with the heading “Development of an Online Assessment System,” there is a reference to open source.

Is ELPA21 requiring the development of an online system that would then be delivered to ELPA21 and become open source code that could be used by other entities similar to the open source online system being developed by Smarter Balanced, or is it allowable for the online system to be an existing system wholly owned by the contractor throughout the contract and following the contract?


16 Table 1 Schedule of Major Deliverables, last row

18 Please clarify exactly what Vendors are expected to provide as “open source” in the RFP requirement, “submit open source system codes to Council.” Is the Council’s expectation that the testing platform to be used for the field test be delivered as an Open Source to the Council at the end of the contract?





Please see the response to #13. 17 20 The RFP indicates there is an ELPA21 item bank. Will all items be

located in the bank so that the Offeror will only need to draw items for the field test from the one source? Also, in what format will the items be provided?

Items will be available in XML that is structured to adhere to QTI v2.1 and APIP v1.0 Standards. All item files will be available in a compressed (.zip) folder.

18 5 20 For the Technology Enhanced items, please provide a list of the different interactions that must be supported (e.g., drag-and-drop, hot spot marking, etc.).

We expect item interactions to include

Choice

Extended text

Hotspot Match

Drag and drop

Inline choice

Text entry 19 5 20 “For each publication, the contractor should submit a development

plan…” Does this mean one plan that outlines development of all materials? Or, does it require a separate plan for each individual material?

One plan that outlines development of all materials is acceptable.





20 5 Functional and Technical Requirements for Field Test Development

20 The RFP states that the successful bidder will be responsible for field testing approximately 3,000 items.

Could you please clarify the division of these 3,000 among items newly developed for ELPA21 by the current item development vendor and those that are state-contributed?

ELPA21 has decided not to field test the items contributed from the states. Approximately 2,100 items will be developed for field test. Please use this number for any related costs.

21 Section 5 Functional and Technical Requirements for Field Test Development

20 (p 20) Forms will be developed according to the specifications provided by the Council in conjunction with the item development vendor.

Is ELPA21 interested in computer-adaptive test solution for the operational test?

This RFP does not include an operational test. A computer- adaptive test solution is not needed for the field test.

22 Secure File Transfer Protocol (SFTP)/Online Directory

20 The contractor will maintain an SFTP site for the receipt and transmission of data files and other deliverables between ELPA21 and the contractor. All such transmissions involving student level data will be done in such a manner as to maintain confidentiality as directed under the Family Educational Rights and Privacy Act (FERPA). The site should be well organized and maintained for the entire length of the contract. Backup and recovery procedures will be

Will participating states be prepared to enter into security agreements with the contractor in order to transfer student- level personally identifiable information (PII) between states and the contractor?

Yes.

a) Will ELPA21 desire to have access to PII and be willing to ensure

specific security agreements to transfer student-level PII between ELPA21 and the contractor, at the states’ direction?

No, ELPA21 as an organization will not have access to personally identifiable information (PII). However, the contractor will provide blind student-level identifiers so that the ELPA21 organization and its research partners have full

http://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html







established and implemented by the contractor. The contractor is required to provide a system that allows for the automatic registration of schools/students using an upload of an Excel spread sheet or a CSV file prepared in a format specified by the Council.

access to all attributes except those that are PII.

23 5. Functional and Technical Requirements for Field Test Development

20 Pool of items from:

ETS and

ELPA21 member states

When will the items from ELPA21 and ETS be delivered to the Field Test Development and Administration vendor?

The item development contractor, ETS, will deliver the ELPA21 item pool on October 31, 2014.

24 20-21, 26-27

For costing purposes, what percentage of the CR items at each grade band and domain are anticipated to be computer-scored vs. human scored?

For bidding purposes, assume that 25% of the CR items scored 0-2 (approximately 100 items in total) are eligible for computer scoring. That is, these items can be scored with 100% accuracy based on specific scoring rules (e.g., table look up of all possible correct answers).

25 5, Table 3 20, 21 Of the 680 CRs listed in Table 3, please provide the number of CRs by Domain that will require human hand scoring.

See response to question 24.

26 5. Functional 20, 21 Table 3 Technology-Enhanced Can you please describe exact types of TE items used within the





and Technical Requirements for Field Test Development

Items ELPA21 test blueprints?

For example, drag-and-drop, selected text, etc.

See response to question 18.

Describe what a Speaking TE item would look like.

Items are still in development, thus no specific description is available. Speaking items will be based on a prompt or stimulus presented to the student; the student will provide his or her response via a microphone, and the response will be captured by the delivery system in a form that allows for scoring after the assessment.

27 5 Functional and Technical Requirements for Field Test Development

21 Does the Council require a demonstration of the APIP functionalities prior to the go-live FT administration date?

Yes. The ELPA21 states want to test out APIP functionality during the field test.

28 Field Test Forms

21 By November 2014, the Council will provide the contractor with a set of Excel worksheets that contain a unique identifier, sequence number, and key metadata (including allowable accommodations) for each item.

Will items also be provided in XML at this time?

Yes.

29 22 Headsets – Should the bidder include costs for the headsets in the proposal? If so, please provide quantities of headsets to be provided. Also, is the contractor expected to ship the headsets to schools?





No, the schools will provide their own equipment for the field test, including headsets.

30 N/A 22 Please confirm that student responses during the speaking test will be recorded and scored by the contractor.

Yes.

31 Accessibility P2

22 For “compatibility with assistive technology devices,” please provide a list of the devices with which the online testing system must be compatible.

Possible assistive technology (AT) devices include an alternative mouse (e.g., trackball), switch control or interface, and a head tracking device. Because AT devices are constantly changing, ELPA21 is not able to provide a complete list at this time. ELPA21 states will be surveyed to determine the AT devices to be available during the field test and provide a recommended list to the field test contractor in June 2014.

32 Field Test Forms

22 The RFP states, “The test window will be several weeks in February and March of 2015 as specified by the Council.”

May vendors assume that the field test window would be all of February and all of March?

Yes.

33 Braille Form 22 One field test form for each grade band should be produced in contracted and/or non-contracted braille, depending on grade level.

Does this requirement include producing Braille forms for K and 1 as these grades normally are not brailled?

It is agreed that braille forms are unlikely to be used for grades K– 2; therefore, ELPA21 is suggesting that braille forms are required only for grades 3 and above. ELPA21 states will develop a plan to ensure the inclusion of ELLs who are blind in the earlier grades.





34 6 24 The RFP refers to “options for how test administrators may certify that they meet minimum requirements to administer the assessment with fidelity.”

Please describe the expectations for test administrator certification. Does the Council envision a certification test that must be passed by administrators?

Yes. One option could be a certification test that must be passed by test administrators, and other options could also be considered. The bidders can suggest options in the proposal.

35 6 Functional and Technical Requirements

25 Are the administration manual and user guide mentioned on page 25 to be used in training the same as the Field Test Administration Manual and Field Test Coordinator’s Manual that are listed in the cost proposal?

Yes. The administration manual and user guide mentioned on page 25 address: (a) the field test purpose, test administration, security features and issues, etc., and (b) step-by-step instructions (with screen shots) for test administrator and student login, specifications about the computer platforms and operating systems supported, and resolutions to common technical problems encountered. These are identified in the cost proposal as “Field Test Administration Manual and Coordinators Manual.”

36 6 Functional and Technical Requirements for Field Test Administration Training

25 Does the Council require accommodated versions, such as Braille and LP forms, of the practice test?

Yes.

37 6 Functional and Technical

25 Does the Council expect translations of instructions (not test content) to be provided in any languages for the practice test and/or





Requirements For Field Test Administration Training

the field test? ELPA21 states are determining whether translated directions will be provided. Please provide an estimated cost with the translation of directions into Spanish in the “Other (specify items)” line on the cost proposal table (page 59).

38 7 26 Please confirm that a full set of training materials is required for each item that requires hand scoring prior to hand scoring.

Yes, that is correct.

39 Help Desk System and Services

26 a. The RFP states that “Tier 1 support includes an email help link and a toll free number.” The same paragraph mentions “chat requests.”

a. Please clarify if vendors should include instant messaging chat capability in the Help Desk Services during and outside of registration and testing windows.

The instant messaging chat is part of Tier 2 Help Desk support during the registration and testing windows.

b. Please clarify the availability needed for Tier 1 Help Desk support outside of the registration and testing windows. The RFP states 7am to 9pm during registration and test administration windows.

Tier 1 Help Desk support should be available during and outside the registration and testing windows.

40 7, 8 26, 30 The RFP states that “[t]he proposal should combine the use of both handscoring and computer-based objective

Please confirm that “computer-based objective scoring” does not refer to “artificial intelligence” scoring of student constructed responses.





scoring.” In a later section the RFP says that “computer-based objective scoring will be used for selected- response items, technology- enhanced items, and some constructed-response items which can produce a look-up list from student responses to the field test items or for which there is a finite, definable set of correct responses.”

Yes, that is correct.

41 7 27 Are third readers required only for writing since each writing response is scored by two readers?

Yes.

42 28 The RFP indicates that the Council will monitor training and scoring. Is the contractor expected to cover the costs of travel for Council members to attend training? If so, please indicate how many Council members will be in attendance at scorer training.

Please plan to cover the costs associated with these meetings for three (3) ELPA21 representatives.

43 7 28 Will the Council provide the “basic certification test which would indicate ‘expertise’ in the areas of reading, writing, speaking, and listening”?





No, the bidder should provide the certification test for the readers.

44 Field test sampling

29 A field test data file must be delivered to the Council according to the schedule, and it must be accompanied by a crosswalk describing data fields that are common in definition and format to PARCC and Smarter Balanced.

Is it expected that the field test data file will be as closely related as possible to PARCC and Smarter Balanced data file specifications?

Yes.


29 Each field test item should be administered to a sample that varies with respect to gender, ethnicity, disability, etc., as well as the length of time since initial enrollment and placement and prior exposure to English (including instruction in English). To facilitate the measuring of student growth in language proficiency over time (within each domain), the samples obtained from each grade band should be as similar as possible in terms of the variables listed above. For example, the sample for each grade band should have the same percentage of students who had no prior instruction in English. This

To what degree will the contractor be responsible if the sample design (e.g., expected N count; expected student participant characteristics) is not achieved within the schedule/scope of the field test study?

The vendor should expect to work with the ELPA21 Assessment Design, Field Test, and Scaling (ADFTS) task management team to develop a sampling plan that would be submitted to the Consortium Council for approval prior to the initiation of field test recruitment/registration. The Consortium Council will consider whether the plan is reasonable (adequate for the purposes intended and feasible/realistic given time and resource constraints) and commit to providing support to achieve the recruitment targets described in the sampling plan (working with/through the consortium states to ensure the necessary levels of participation). If the vendor encounters any challenges to achieving sample requirements and participation, the vendor should communicate with the Council and work with ADFTS team to resolve all issues.





approach is not meant to provide a sample representative of the actual ELL population but rather one that is representative of a synthetic reference population


29 Same as above Will ELPA21 and/or states provide the ELPA21 states’ student characteristics data (i.e., gender, ethnicity, disability, etc., as well as the length of time since initial enrollment and placement and prior exposure to English (including instruction in English)) to the contractor for use to develop the 4sample design?

Yes.

47 8 Functional and Technical Requirements

29,32 Do Consortium-member states have an obligation to provide field test participants? Is the contractor responsible for the field test participant recruitment or will ELPA21 recruit for the field test? If the contractor is responsible, will the Council assist with the recruiting efforts for the field test and pilot test? Will there be monetary incentives for field test participants, and if so, what will the incentives be and who will be responsible for them?

The sampling and recruitment for the pilot study and field test will be the responsibility of the selected vendor, with the support of ADFTS TMT and the member states. The ADFTS TMT and/or member states will help in obtaining contact information and profile information (e.g., location and distribution of the ELL population), providing supportive communication, and troubleshooting when recruitment issues occur.





There will be no monetary incentives for field test participants. 48 8 Functional

and Technical Requirements

32 Will the items for the Pilot Study be drawn from the same set of items as the summative assessment and the screener?

Yes.

49 Pilot Study 32 Does the Council require the vendor to score the pilot items? Is there a minimal student sample size that the consortium is expected to have?

Yes, the vendor is required to score the pilot items. There is no minimum number of participants in the pilot study (either overall or per item). The vendor should propose the design and the number of participants needed to accomplish the goals of the pilot test. Proposers should describe the methods that they expect to use in the pilot study in the proposal.

50 8 Functional and Technical Requirements for Assessment Research and Reporting

32 The Council is requesting a pilot study in January 2015. Is the administration of the pilot study limited to PC’s, or will this study include notebooks and tablets?

This study will include PCs, notebooks, and tablets.

51 33 The RFP indicates that the contractor is responsible for costs for the TAC meetings at 1.5 days each, twice per year. Given that this is an 18 month contract, how many meetings should be budgeted?

Three TAC meetings.





52 9. Functional and Technical Requirements for Online Assessment System: Hardware and Software Requirements

36 6 Software Functional Requirements

Required Functionality: An automated interface allowing the ELPA21, or designee, to import test items in various formats (e.g., multiple-choice, constructed response, technology-enhanced, etc.) from ELPA21’s item bank. Batch importing via XML must be available as well.

For some of the more complicated TE response types, QTI does not have a native representation of the item.

Does the item development vendor plan on providing all items in a QTI format?


Will the items from the States also be delivered in a QTI format?

ELPA21 has decided not to field test the items contributed from the states.

53 Cost Proposal 59 May vendors provide optional services and/or studies and cost line items for the Council to consider?

Yes. Please create a separate cost proposal.

54 Cost Proposal 59 Please clarify which cost line vendors should use for shipping and retrieval costs of braille and LP forms?

The cost for shipping and retrieval costs can be included in the Braille Form and Large Print Form lines.

Documents

English Language Proficiency Assessment for the 21st ... Field Test RFP_Questions... · English Language Proficiency Assessment for the 21st Century (ELPA21) Oregon Enhanced ... Number