NHES 2012 Pre-Proposal ConferenceJanuary 19, 2011

Overview• History and purpose of NHES• Redesign• NHES 2012• Overview of data collection activities

– Sampling, questionnaires, data collection procedures, telephone non response follow-up and data file creation

• Overview of ED / NCES procedures– Deliverables and review procedures– Security plan and data protection

• Question and answer

• Designed to study education research issues that cannot be addressed efficiently in institutional surveys

• Allows NCES to collect data on a wide range of topics directly from households, including nonparental child care, parent involvement in education, school readiness, and adult education

• Designed to allow for the analysis of trends in a number of important topics over time

History and Purpose

• Nationally representative samples of children from birth through grade 12, as well as adults

• Each administration has a screener survey and 2 or 3 extended, topical surveys

• Had been conducted by telephone using list-assisted random digit dial (RDD) design

• Interviews conducted in English or Spanish

• Data collected from January through May because of school related items

History Continued

Redesign• In 2008, a contract was awarded to redesign the

NHES • There were two phases:

– A small feasibility test in 2009 (pilot test)– A large-scale field test starting in January 2011 (field test)

• The final design will be guided by results from the 2011 field test– The SOW will include options to account for the possible

final design

NHES 2012 Overview

• Data collection begins January 2012 and ends May 2012

• Two-phase (screener and topical) mixed-mode (mail and telephone) data collection

Data Collection• Households will be screened by mail for eligible children• Three topical surveys will be fielded by mail:

– Early Childhood Program Participation (ECPP)• Targets children from birth through age 6, not in kindergarten

– Parent and Family Involvement in Education (PFI)• Targets children age 6 – 20 in kindergarten through 12th grade (or equivalent); • includes a related but distinct homeschool questionnaire

• Spanish versions of all instruments will be available (a bilingual screener may also be offered)

• Surveys have already been designed but may need slight modifications and updates

NHES 2012• Address Based Sample (ABS) will be utilized

– Sampling of addresses from a near universal listing of residential mail delivery locations

– Frame should be based on most recent USPS- DSF data– Contractor will need to provide a plan that details how they will

handle frame issues such as throwbacks, vacant units, P.O. boxes, etc. during sampling, and PMRs and other issues during data collection

– A phone match will be required for telephone non response follow-up

Sampling• The final sample must be nationally representative of

the two target populations:  1)  children 0-6 and not yet in kindergarten and 2) children in grades K-12

• Oversampling will likely be required to ensure adequate Black/African American, Hispanic and other minority group representation for analysis

• Contractor will need to develop a plan for within household sampling on a rolling basis

Sampling Continued

• We estimate starting sample of approximately 198,000 households– Contractor can suggest alternate sizes– Carefully consider the design aspects of the

survey when projecting response rates and eligibility rates

– Sample size is driven in part by precision requirements for estimates

Screener Data Collection

• Initial screener mailing will contain a $2 or $5 cash incentive depending on results from 2011 field test. – Security procedures will need to be in place for

handling cash incentives (mailout and return)– A magnet may also be included– A prenotice letter prior to the initial screener

mailing may also be used

Screener

Non Response Follow-up: Screener• Up to three screener questionnaires will be mailed to

households– A possible prenotice letter – Reminder postcard– Third questionnaire mailing will use FedEx or Priority Mail– All packages include: cover letter, questionnaire, and

prepaid return envelope

• Telephone non response follow up may be utilized depending on results from the 2011 Field Test

Non Response Follow-up: Topical• Up to three topical questionnaires will be

mailed to households– First mailing may be sent by FedEx or Priority Mail

or include a cash incentive if using first class mail– An alternate envelope (color and wording) from

the screener envelope may be required– All packages include cover letter, questionnaire,

and prepaid return envelope

• Option for telephone non response follow-up

Information to identify child will be printed on form

We would also like to customize items

School Information Will be Printed from CCD and PSS files on PFI

Contractor will need to identify schools in the household’s neighborhood from the CCD and PSS then print the names of these schools on this item. Respondent will mark which school their child attends. Contractor will later merge data about the selected school onto the restricted use data file from the CCD or PSS.

Other Considerations

• Across all three screener waves 472,000 questionnaires could be mailed

• Across all three topical waves 86,000 questionnaires could be mailed– Contractor should demonstrate experience managing

mailing operations of this size and complexity

• Quality control is critical– for privacy/confidentiality reasons questionnaires can not be

sent to incorrect household (would be a reportable privacy breach)

Options• Screener switch

– Testing two versions of screener (screenout and engaging) in 2011. Looking at impact of ‘switching’ version at 1st non response follow up.

• Testing impact of prenotice letter for first screener mailing

• Testing a variety of topical contact strategies– Differing incentive levels, mailing envelope,

special handling

Telephone Non Response Follow-up• Telephone underperformed compared to mail in

2009 Pilot• Testing as a non response follow-up mode in

2011• If option is exercised, looking for most cost

effective implementation• Screener and topical are separate events • Control system needs to be able to move case

back and forth between modes

Telephone Non Response Follow Up Continued

• Interviews conducted in English and Spanish • All cases will need to be treated as refusal

conversion• Information that will be helpful:

– Number of experienced interviewers available for NHES

– Overall phone response rates from past national studies you have conducted

Telephone Considerations• Approximately 32,000 screener non response cases

– Roughly five week period to work screener cases before switching to topical

• Approximately 7,000 topical non response cases (PFI and ECPP)

• In past NHES ~5% cases completed in Spanish• Topical instrument will need to display data from screener and

have a school look up table to match mail forms• Contractor should demonstrate experience managing a

telephone operation of this scale and complexity within the NHES schedule

Telephone Number Look up• The Contractor should plan to look up telephone

numbers for mail non responding cases where there is no match on the sample frame or the match is incorrect. (Sometimes called tracing or research)– Vendor databases used for searching need to meet NCES

guidelines• E.g., vendor’s database can not be updated using NCES data and all

information must be securely deleted after search is complete • A data protection contract must be in place between contractor and

vendor doing the matching

Data Collection• Screeners must be QC’d and entered quickly to prepare for

topical mailing– If names are collected, they need to be reviewed to ensure only valid names

are printed on topical (ideally through some combination of manual / automated review).

– Invalid names consist of curse words, blatantly fake names, etc.

• Topicals will be mailed on flow basis no more than three weeks from receipt of screener

• Less than 21 days to process screener, sample child, prepare topical print file, print (or label) topical questionnaire and cover letter, and mail

Metadata• Contractor will need to collect and report

metadata on all aspects of study– Data needed to evaluate effectiveness of different

collection stages and modes, efficiency of case management» E.g., date questionnaire mailed, date received at

contractor, questionnaire wave, date keyed, respondent call logs, postmaster returns, case status including call attempts and outcome codes.

– File preparation:• Flag variables edited or imputed, rate at which autocoding did

not work and number of missed skip patterns

Data Entry

• Contractor will be responsible for validating the accuracy of the data collected from all modes of data collection

• Whatever method of data entry is selected by the contractor should already be validated for accuracy on large scale questionnaires with text, numeric and closed response options and have built in QC measures

System Design

• In proposal contractor should emphasize their approach and experience handling the following aspects of data collection:– Checking in and processing large quantities of

questionnaires that are returned in waves • Questionnaire design and design software capabilities• Data entry approach and ability to process these quantities of

questionnaires within the specified time frames• Ability to track and control individual forms from printing stage

through archiving

Building and managing case control systems for two phase multi-mode studies

• The short data collection time frame and complex design necessitate an automated case management system– Project Manager should have daily access to

sample performance in case course corrections are necessary

Case control systems for multi-mode studies continued• Requests for additional paper questionnaire:

– From respondents during phone non response follow-up:• Contractor needs to generate a new mailing package,

mail the questionnaire within a couple of days and place a two week hold on call attempts to the household to allow time for response.

– From the government:• The government may request an additional mailing to

households with certain phone status codes (e.g., soft refusal on the phone or ring no-answer).

Data File Creation• End product will be separate ECPP and PFI

datafiles (restricted and public versions). Some screener data will be carried over to the datafile

• Contractor will draft and implement specifications to edit and impute data– Should have experience editing mail survey data

Data File Creation - Editing

• 100% imputation of missing and out of range values– NHES has used ‘hot deck’ procedures in the past

• Should have experience developing and implementing hot deck imputation specifications

• No unreadable fields or unallowable characters• Valid skips are not imputed and can be distinguished

from valid zeros and nonzero data• Files are checked for internal consistency and basic

tables are run and reviewed• Duplicate responses are reconciled

Data File Creation- Composite & Classification Variables• Contractor will create 20-30 composite or

classification variables – SES using income measures, parent’s education, count of

adults in HH, etc.– Some variables pull from external sources (ACS, Census,

Common Core of Data)

• Variables should be consistent with past NHES• All code documented and included in manual

Data File Creation• Prepare for Disclosure Review Board (DRB)

– Prepare a Disclosure Analysis Plan (DAP)• A plan that proposes how the data will be protected from disclosure. This may

include topcoding and perturbing data as well as other disclosure avoidance techniques

• Weighting to ACS and Census– Create person level weights, rake to population totals

• Standard Errors– Complex sample design, provide replicate weights and base weights– Previous NHES studies use Jacknife methods and PSU / Strata variables

for Taylor Series variance estimation• Design effect values must be included in documentation• NCES seeks to release files within 8 months of data collection

Data File Creation

• SAS, SPSS and Stata versions• Files prepared for upload to EDAT (NCES online data

access tool)• Codebooks that include a listing of each variable, its

values, labels and unweighted and weighted frequencies

• User’s manuals (see examples on our website)– PFI, ECPP, overall and restricted use manuals

• Cover all aspects of the survey design, sampling, data collection, data anomalies, data file structure and layout, and weighting

Reports and Review

• NCES thoroughly documents datasets– Fully documented sampling plan, weighting plan,

edit specifications, quality control procedures– Conduct bias analysis

• Using frame and external data and possibly mode / wave data from the file

– Accurately document response rates and case outcome codes

Definition of “Draft”

• Complete product of high quality • Should be fully formatted and free of

grammatical and typographical errors• The government expects that only minor

revisions should be needed for drafts to become final deliverables

Reports

• All reports must strictly adhere to NCES guidelines for style and content

• A draft will not be acceptable if it does not meet these guidelines

Review Timeline for Publications

1) COR review (2 weeks)

2) Management review (3 weeks)

3) Senior management review* (6 weeks)

4) Technical review* (6 weeks)

5) Commissioner review (4 weeks)

6) IES Review (6 weeks)*concurrent

Security Considerations

• Contractor needs to have facilities to securely store over 250,000 questionnaires

• Sample information, as well as questionnaires, may be considered Personally Identifiable Information (PII) – Attempts should be made to minimize the number of

subcontractors needing access to this information– All subcontractor employees and systems that will come in

contact with NCES PII will need to meet the same requirements as the prime contractor, including background investigations and information systems certification and accreditation

Additional Security Considerations

• If using an outside vendor for the sample frame, steps will need to be taken to minimize disclosure risk (e.g., drawing additional cases so vendor will not know who is in final sample)

Data Security Plan

• Identify all processes where PII is collected, handled, and transferred

• Explain how PII will be protected during all phases of collection, handling, and transfer within your staff and information systems, as well as those of any subcontractors

Information System Security• Contractor and subcontractor will need to go through

Certification and Accreditation (C&A) process for their computer systems and receive an Authority to Operate (ATO) before data collection can begin

• Extensive documentation and on site visits are required for this process

• Contractor should allow 3 months from submission of complete paperwork to receive ATO

Contractor Should be Familiar With and Able to Meet the Requirements of:• IES Confidentiality Statute (20 U.S.C. 9573)• Confidential Information Protection and Statistical Efficiency Act

(CIPSEA) of 2002 (Public Law 107-347, Title V)• Federal Information Security Management Act (FISMA) of 2002

(Public Law 107-347, Title III)• Management of Federal Automated Information Resources (OMB

Circular A-130; Appendix III)• OMB Memorandum: M-06-15, M-06-16 and M-06-19• National Institute of Standards and Technology (NIST) standards

(NIST publications 800-37, Rev. 1; 800-53 Rev 3; 800-60)• FIPS Publications 199 and 200

C & A Process and Timeline for New System– Preliminary C&A Activities

• e.g., Project Kick-Off Meeting, Data Sensitivity Worksheet, System Categorization (FIPS 199)

– Phase I – Creation of Security Documents• System Security Plan (SSP), Contingency Plan (CP), Configuration Management Plan (CMP)

– Phase II – Conduct Security Assessment (3 to 4 weeks)• e.g., Risk Assessment Report, ST&E, POA&M, Executive Summary, C&A Memo

– Conduct Final Security Assessment Review (FSAR) (4 to 6 weeks)• Report detailing the current security posture of the system• Provides certification recommendations for the system• Findings from ST&E need to be addressed by System Owner. IV&V will work

with System Owner to either 1) remediate, 2) accept risk, or 3) create a corrective action plan (CAP) for each finding, including completion dates

– Accreditation of the System (1 week)• Preparation and signing of the Authorization to Operate (ATO) Letter

– C&A Start-to-Finish Time (12 to 17 weeks)– Re-Certification of previously C&A’d systems may take less time, depending

on current state of documentation

43

Timing Considerations• Many complex documents will need to be

created concurrently after contract award to meet schedule dates: – C&A documents, Data Security Plan, cover letters

and questionnaires, sampling plan, project schedule, and OMB package are all needed within approximately the same time frame

• NCES seeks to release data within 8 months of data collection

Questions?

Contact Information

• Sharon Masciana • Contract Specialist • 550 12th Street, SW, Room 7129 • Washington, DC 20202-4240 • Office - 202-245-6132 • Fax - 202-245-6296 • Sharon.Masciana@ed.gov