Predictive Analytics in Child Welfare - ASPE...Predictive analytics is a set of advanced analytical methods that may enable child welfare agencies to leverage a range of case -level

Predictive Analytics in Child Welfare Considerations in Contracting Vendors for Predictive Analytics

April 2018

By Christopher Teixeira, Michelle M. Jacobs-Smith, and Matthew Boyas

The MITRE Corporation

This report was prepared by staff of the MITRE Corporation as operator of the CMS Alliance to Modernize Healthcare Federally Funded Research and Development Center under a project sponsored by the Assistant Secretary for Planning and Evaluation. The findings and conclusions of this report are those of the authors and do not necessarily represent the views of ASPE or HHS. Approved for Public Release; Distribution Unlimited. Case Number 18-1412.

ii

Executive Summary Predictive analytics is a set of advanced analytical methods that may enable child welfare agencies to leverage a range of case-level data about families’ situations, turning hindsight into insight and improving child welfare outcomes. Some child welfare agencies are currently implementing predictive analytics, though these efforts are in their infancy. In most existing cases, child welfare agencies decided to partner with an outside organization to run the technical analyses, as opposed to building in-house statistical or predictive modeling expertise.

This document: • outlines sections for a request for proposal (RFP) for an organization to procure predictive analytic

capability, • provides guiding principles and ideas that should be considered during the contract development

process, and • walks through examples specific to the context of child welfare.

Every child welfare predictive analytics project is unique; this document is not meant to be a complete “fill-in-the-blank” template. Agency priorities and stakeholders are different, and it is unreasonable to expect the suggestions presented here will be relevant for every RFP. These guiding principles are intended to inform agency staff considering contracting for development and/or implementation of a predictive analytics model applicable to child welfare services.

Many sections of an RFP are boilerplate and require nothing specific to predictive analytics or child welfare. In contrast to these standard RFP sections, three RFP sections can be tailored to predictive analytics applications intended to improve child welfare outcomes.

While benchmark language can be included in such sections, these sections are best written to include nuance and details specific to the proposed predictive analytics project. This document focuses on these more complex aspects of the acquisition package for predictive analytics.

For convenience this document uses terminology standard to federal contracts. Most states have developed acquisition policies and manuals which loosely follow Federal acquisition practices to guide their contracting and acquisition activities. While a specific agency may not follow this contracting model exactly, the content discussed in these sections should be represented somewhere in the agency’s format and can be adapted to a specific situation.

Table 1: Targeted RFP Sections

Section Name Description Requirements Documentation Defines all work performance requirements for an offeror

and facilitates the preparation of a proposal Special Contract Requirements Specifies any special provisions, terms and conditions

Evaluation Factors for Award Outlines the evaluation scheme that the agency will use to select the winning proposal

U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation

iii

Contents Introduction to Predictive Analytics in Child Welfare ..........................................................................1 Types of Predictive Analytics RFPs ......................................................................................................2 Targeted RFP Sections ........................................................................................................................4

Requirements Documentation ........................................................................................................ 4 Constraints .......................................................................................................................... 4 Assumptions ........................................................................................................................ 5 Requirements ...................................................................................................................... 6

Special Contract Requirements........................................................................................................ 6 Special Contract Requirements for Predictive Analytics .................................................... 6

Evaluation Factors for Award ........................................................................................................... 8 Evaluation Factors vs. Evaluation Criteria ........................................................................... 9 Evaluation Criteria for Predictive Analytics ...................................................................... 10

Summary ......................................................................................................................................... 13 Appendix ......................................................................................................................................... 14

Standard RFP Sections ................................................................................................................... 14 Acronyms ....................................................................................................................................... 16

1

Introduction to Predictive Analytics in Child Welfare Predictive analytics is a set of advanced analytical methods that may enable child welfare agencies to leverage a range of case-level data about families’ situations, turning hindsight into insight and improving child welfare outcomes. Predictive analytics employs statistical techniques to discover patterns from data about the past and present and make inferences about future behavior or events.1 Some child welfare agencies are currently implementing predictive analytics, though these efforts are in their infancy. In most existing cases to date, child welfare agencies have decided to partner with an outside organization to run the technical analyses, as opposed to building in-house statistical or predictive modeling expertise. For more information on efforts going on around the U.S., see Predictive Analytics in Child Welfare: An Assessment of Current Efforts, Challenges and Opportunities.2

Predictive analytics models are best applied in situations where something that will happen at some point in the future—such as a placement disruption or foster care re-entry—occurs frequently enough that statistical models can be built within acceptable tolerances. However, even when predictive analytics can be applied to a specific problem from a technical perspective, other attributes of the project, such as cost, implementation strategy, population size, and model accuracy, may steer an agency away from pursuing it. For more information on criteria for determining if predictive analytics is appropriate for a specific problem, see Predictive Analytics in Child Welfare: An Introduction for Administrators and Policy Makers.3

While engaging an offeror for a predictive analytics project may be an easier path forward than hiring data scientists, developing a comprehensive and appropriate request for proposal (RFP) that sets the project up for ultimate success is a challenging task. This document is a guide to issuing an RFP for a predictive analytics project in the context of child welfare. The information here is meant to inform RFP development and can be adapted to an agency’s particular situation and goals. As each child welfare agency and every child welfare predictive analytics project is unique, this document is not meant to be a complete “fill-in-the-blank” template. It is important to remember that every agency’s problem, structure, and other driving dynamics are different, and there is no one size fits all RFP. Therefore, this document outlines a predictive analytics RFP, provides guiding principles and ideas that should be discussed as an agency determines its requirements, and walks through examples specific to the context of child welfare.

1 Definition of predictive analytics adapted from Gartner, Inc. (2016). Available at: http://www.gartner.com/it-glossary/predictive-analytics-2/. 2 Available at: https://aspe.hhs.gov/predictive-analytics-child-welfare. 3 Available at: https://aspe.hhs.gov/predictive-analytics-child-welfare.

https://aspe.hhs.gov/predictive-analytics-child-welfare



http://www.gartner.com/it-glossary/predictive-analytics-2/

http://www.gartner.com/it-glossary/predictive-analytics-2/




2

Types of Predictive Analytics RFPs When writing an RFP for a predictive analytics project in child welfare, child welfare agencies can typically take one of two paths: research-focused or software design. Research projects are generally oriented at exploring the feasibility of predictive analytics to the specific child welfare agency’s data and systems. Alternatively, software design projects attempt to build a production-ready tool that allows predictive analytics results to be used by agency staff.

The decision between issuing an RFP for a research project or a software design project—or some combination of the two—is largely a strategic one. The two types of efforts ask different questions and accomplish different goals, and the choice between research or software design would depend less on characteristics of the acquisition and more on the child welfare agency's priorities, requirements, and desired outcomes. For example, an agency that is looking for a new risk screening tool to deploy quickly would likely issue a software design RFP, perhaps purchasing a license for an existing commercial off-the-shelf (COTS) tool to reduce design cost. Alternatively, an agency that has perhaps experienced a larger-than-normal number of child fatalities and would like to understand the driving factors may embark on a research project to explore the characteristics in common between cases; this research project may ultimately develop into software design where the research results are formalized and integrated into a piece of software for caseworkers to use.

It should be noted that these two categories are extreme generalizations, and it is certainly possible for an agency to develop an RFP that is part research-oriented and part software development. These two categories are briefly summarized in Table 2, below, and are discussed in more detail in the paragraphs that follow.

Table 2: Types of Predictive Analytics Projects


3

A research project would, in most cases, be focused on building a proof of concept solution. Offerors would need one-time access to proprietary—and likely sensitive—data, but they would not have the need to integrate those results back into a production-ready system hosted by the child welfare agency. In addition, a research project might utilize sensitive person attributes such as gender, ethnicity, or age, with all identifying information removed, significantly reducing potential privacy concerns. Research projects produce code to answer the research question, but implementing this code into production systems can be difficult and is out of scope for a research project. Research projects also will often be much more self-contained, in that once the offeror is provided data, that vendor can often run the predictive analytics research without significant agency involvement. The final product of a research project can be a publicly-released publication of results, which is often accompanied by a set of other considerations regarding the release of information.

In contrast, a software design RFP would often take the proof of concept from a research project and then extend that out to a full, real-time implementation within the child welfare agency’s procedures. Such an RFP has different aspects to consider, such as ensuring integration of analytical results with existing systems and building out appropriate policies for the use of the new information. Production systems by necessity retain child and family identifiers needed for use by agency staff to link data systems. Furthermore, predictive analytics software solutions will require maintenance, both for the software itself as well as refreshing the models as new data become available. Because an RFP of this scope would fundamentally change the child welfare agency’s work process and systems, the RFP could involve many more agency stakeholders to ensure the best chances of success.


4

Targeted RFP Sections Several sections of an RFP are essentially boilerplate and do not require anything specific to predictive analytics or child welfare. However, three sections of the RFP should be tailored to accommodate specific predictive analytics characteristics. For more information on the boilerplate standard sections, please see Appendix A. This section will provide a detailed look at the three more complex sections that are specific to predictive analytics applications: Requirements Documentation, Special Contract Requirements, and Evaluation Factors for Award.

Most states have developed their own acquisition policies and manuals which loosely follow Federal acquisition practices to help guide their contracting and acquisition activities. While a specific agency may not follow this contracting model exactly, the type of content discussed in these sections should be represented somewhere in the agency’s format and can be adapted to a specific contracting situation.

The following sections of this document will outline the characteristics of these targeted sections and highlight ways in which they may be customized to address the specific needs of predictive analytics projects. While benchmark language could be included in the three targeted sections, these sections will have nuance and details specific to the predictive analytics project that the child welfare agency is initiating. The following information is intended to help craft that nuanced and detailed language.

Requirements Documentation Work statements enable offerors to clearly understand the agency’s motivation behind seeking a proposal to develop or produce the specified goods or services. They define (either directly or by reference to other documents) all work performance requirements for an offeror. They also facilitate the preparation of a proposal and aids the agency in conducting source selection and contract administration after award. To achieve this, the agency can develop a Statement of Work (SOW), Project Work Statement (PWS), or Statement of Objectives (SOO). To minimize risk to the agency, it is important that requirements are stated to address the “outcome” rather than the “how to.”

A clear and concise work statement permits the agency and offeror to estimate the probable cost and the offeror to determine the levels of expertise, manpower, and other resources needed to accomplish the task. A work statement (SOW, PWS or SOO) is typically the most involved section of an RFP which includes the scope or a description of the work. The work statement is typically broken down into three subsections that adequately encompass this necessary information. These subsections are described in sections 3.1.1 to 3.1.3. The examples described here are not in any way exhaustive but are included to illustrate the types of concerns specific to predictive analytics work that may be addressed in each of these key sections.

Constraints

This is an itemized list of restrictions within which the offeror has to work. Constraints can vary but might include, for instance, specifications regarding how data should be accessed or stored, the systems or software available to use for analysis, and methods for communicating results.


5

Given their highly technical nature, predictive analytics projects will likely have a detailed constraints section. Offerors will have access to sensitive child welfare data, and appropriate data security practices must be observed. The agency would use the constraints section of the SOW to explain their data security requirements, including how the data is to be transmitted to and from the offeror, what type of computing system the offeror must use to store the data, etc.

EXAMPLE CONSTRAINTS

• The offeror shall ensure the predictive analytics model is compliant with Institutional Review Board (IRB) requirements.

• The offeror shall comply with agency standards for securing data and integrating into the existing agency software.

• The offeror shall manage Personally Identifiable Information (PII) data within a secure network in accordance with agency regulations.

Assumptions

The assumptions section of the SOW is an itemized list of things the offeror can assume will be done or provided by the agency.

EXAMPLE ASSUMPTIONS

• The agency will provide a secure environment that can serve as a central workspace for project documents (statements of work, deliverable documents, project plans, etc.) and team contact information.

• The agency will provide a subject matter expert for consultation on the data sources used for this project.

• The agency will provide all documentation necessary for maintaining management of sensitive data files, adhering to internal file structures, and incorporating into agency workflows.

• The agency will work collaboratively with the offeror to ensure all staff assigned to the project are trained in data privacy issues and able to uphold security and ethical requirements for this project.

• The agency will acquire, maintain, and support any software and hardware relating to the storage, cleansing, and analysis of data for the purpose of this project.


6

Requirements

A requirement is a singular documented need—what a particular product or service should be or how it should perform. It is a statement that identifies a necessary attribute, capability, characteristic, or quality of a system in order for it to have value and utility to a user. A good requirement should be clear, unambiguous, and testable. When writing requirements for an SOW, words such as “assist” or “support” should be avoided unless you are in fact requesting assistance for work the agency will perform. For an SOW/PWS, verbs such as “provide” or “perform” are used when an offeror is performing work for agency review.

EXAMPLE REQUIREMENTS

• The system must be able to perform predictions of future abuse with an agreed upon threshold for accuracy, false positive prediction rates, and false negative prediction rates and in near real-time upon request. These thresholds will be determined prior to performing the analysis in agreement with the offeror.

• The system will integrate with the agency’s existing data warehouses, pulling data in once a day to predict which children are at a high risk for abuse in the next six months.

• The system will assist agency leadership in understanding how the expected rate of attrition will impact the state of its workforce over the next six months and predict the impact to caseload levels for each caseworker.

Special Contract Requirements The special contract requirements section contains any special provisions, terms, and conditions not included in a Contract Clauses section. It can contain clauses that are written specifically for the procurement. The contracting officer has several hundred clauses and provisions to choose from when drafting this section. Every clause included in this section must be included for a reason—either a regulation requires it, or the administration of the contract necessitates it.

However, if the agency is not deliberate about the clauses included in this section, they could unnecessarily add to the contract cost. Furthermore, as the agency issuing the RFP, acquisition personnel must be prepared to answer concerns regarding provisions.

Special Contract Requirements for Predictive Analytics

With respect to predictive analytics projects, there are certain requirements that should be included within the RFP. Depending on the focus of the RFP, the agency should consider the product produced or delivered and how that transition of final work product should occur. For example, if the RFP is research focused with the aim of developing a predictive analytics code base, this clause would outline the process


7

for delivering the code base in a manner that is useful to the agency including the requirements needed for execution of the code.

When it comes to determining when the offeror has adequately completed its obligations under the contract, every agency would like to have a clause that outlines thresholds for model performance. However, setting a threshold for accuracy at 100% (as in the model would need to correctly predict 100% of the time) is unrealistic—no model will ever be that accurate, as all models are an approximation of real-world processes that have some unexplainable error. Instead, a requirement should outline the ability to measure the model’s improvement on performance and consider restricting its implementation if it does not meet certain thresholds.

EXAMPLE SPECIAL CONTRACT REQUIREMENTS – AGENCY

The agency shall:

• Receive from the contractor a listing of the datasets the contractor desires to include in the work. This listing will include descriptions of the necessary data fields and a justification for each field’s inclusion into the analysis.

• Provide the contractor with de-identified data that encompasses a date range.

• Protect the confidentiality and privacy of the dataset(s) and store the data in a secure network folder with limited access.

• Create and maintain the cypher for re-identifying the cases.

• Maintain ownership of the predictive analytics model, its results, and the consequences of using this model once integrated into existing systems.


8

EXAMPLE SPECIAL CONTRACT REQUIREMENTS – CONTRACTOR

The contractor shall:

• Provide a secure location to store the data in accordance with the agency’s Privacy and Security Standards requirements.

• Provide the agency with a description of the methodology used to create the predictive analytics model, including the use of the data provided by the agency and any other sources of data used throughout the modeling process.

• Provide the agency with documentation on processing data for the predictive analytics model, including any methodologies and their results for validating and verifying the model.

• Provide the agency with the expected uses and limitations of the model.

• Provide a bibliography of the literature review undertaken for this project.

• Obtain prior written agreement from the agency to conduct any research using this data with external parties including, but not limited to, personnel of any sub-researcher, research assistants, other research staff members, and research affiliates that are not associated with the research projects identified in the RFP.

• Provide the agency with a copy of any white papers developed in the course of this collaborative project and provide periodic updates to the agency, as mutually agreed upon. White papers shall not include personally identifying information. Any published reports shall provide acknowledgement of the agency role in this project.

• Provide the agency two (2) weeks to review and suggest edits to any and all published reports based on this project.

• Provide the agency with the detailed model performance results compared to benchmarks from the literature (where appropriate) to facilitate the agency’s review of the technical analyses.

• Destroy any links to confidential and/or protected health information and identifiers at the earliest opportunity after data analysis is complete and research is done. The date and method of destruction must be declared to the agency within 5 business days of destruction.

Evaluation Factors for Award Evaluation factors are used to determine which of several competing proposals submitted in response to an RFP would best meet the agency’s needs. The evaluation factors are supported by criteria that are used to determine the offeror’s ability to meet requirements identified in the work statement. Through the evaluation criteria, the agency is able to determine the strengths and weaknesses as well as differentiate the benefits and risks among proposals. A well-integrated evaluation scheme provides consistency, discipline, and rationality to the source selection process.


9

Federal Acquisition Regulations (FAR) requires agencies to evaluate quality, past performance, and price or cost. State and local contracts are likely to vary on this issue. Within federal contracting, quality is commonly assessed using technical and management factors. Secondly, technical and management sub-factors are developed to detail the specific areas to be evaluated. It is important to structure sub-factors to clearly reflect the objectives of the acquisition, user requirements, and perceived risks. Lastly, the factors and sub-factors are listed in their order of importance. The contract specialist develops price/cost instructions based on the type of contract and the Independent Government Cost Estimate.

Evaluation Factors vs. Evaluation Criteria

Before proceeding with detailed examples, it is worth differentiating between evaluation factors and evaluation criteria. Evaluation factors define specific areas of the proposal to be evaluated, while evaluation criteria establish a framework for the agency’s evaluation of an offeror’s proposal. Evaluation criteria will be defined within specific evaluation factors and are specifics on which the proposals can be evaluated for compliance to requirements.

The development and use of standards is the key to uniform application of evaluation criteria. Standards establish the minimum level of acceptability for a requirement and provide the basis on which the ratings above and below the minimum level are set. Stated another way, a standard is the measurement baseline that will be used by the agency evaluator to determine whether a proposal meets, exceeds, or fails to meet a solicitation requirement. Standards, by providing a consistent and uniform measurement target, promote an objective evaluation of proposals.

Examples of evaluation factors and associated criteria are:

Table 3: Example Evaluation Factors and Associated Criteria

Evaluation Factor Evaluation Criteria Price/Cost Factor: Cost or price must be evaluated in every source selection. Contracts can only be awarded at costs or prices that the agency determines to be reasonable.

The agency will use price analysis to evaluate the prices, not only to determine whether the price is reasonable, but also to determine whether the offeror understands the work and has the ability to perform the contract. The Price Proposal will be evaluated for the base period and all optional periods of performance. The Price Proposal will be evaluated to determine completeness (acceptability and validity of the pricing methodology). To assist in determining reasonableness, evaluation of an offeror’s proposal may include comparison to competitor and incumbent rates and other market data. Proposed prices will not be assigned numerical weights and will be subordinate to technical considerations in determining successful offerors. The Contracting Officer will also conduct an appropriate price analysis in accordance with applicable regulations.


10

Evaluation Factor Evaluation Criteria Technical Factor: Measures technical excellence and management capability. When appropriate, this factor will measure corporate capability separately.

The proposal will be evaluated to determine the extent to which the proposed approach is workable and the end results achievable. The proposal will be evaluated to determine the level of confidence provided the agency with respect to the offeror's methods and approach in successfully meeting and/or exceeding the requirements in a timely manner.

Past Performance Factor: Past performance evaluation validates statements made in the offeror’s proposal, motivates offerors to strive for excellence, recognizes good performance, and reduces risk and oversight.

The agency will conduct a performance risk assessment based on the quality, relevancy, and recency of the offeror's past performance, as well as that of its major subcontractors, as it relates to the probability of successful accomplishment of the required effort.

Evaluation Criteria for Predictive Analytics

For predictive analytics projects, each of the evaluation criteria needs to be adjusted or considered differently due to each project’s potential complexity. In terms of experience, the offeror should demonstrate that it has applied predictive analytics to similar problem sets and delivered a product that meets a level of quality expected by the agency. This previous experience does not necessarily have to be within child welfare practice, but the offeror should clearly express the similarities in their previous applications of predictive analytics and how it relates specifically to the problem brought forth by the agency.

The described technical approach needs to clearly define the risks commonly experienced when applying predictive analytics. Common areas of risk for predictive analytics projects involve obtaining access to data (whether de-identified or sensitive) and the potential iterative modeling methodology to improve results of predictive analytics algorithms. These two areas provide risk to both time and cost of a project. A delay in getting access to data will ultimately add time to getting results but typically with little impact to cost. Building a predictive analytics model is inherently risky due to the uncertainty around producing results that meet previously-agreed-upon thresholds. This fact can introduce the need to build multiple predictive analytics models if the first model does not meet contractual requirements. In addition, the technical approach needs to clearly define aspects of predictive analytics that are important to the agency. An example of something that may not meet the agencies needs is using algorithms that are not transparent in their use of data or producing a result that cannot easily be integrated into the agency’s business process.


11

The management approach to predictive analytics is similar to other software-based requests for proposal. Ideally the management approach covers topics like dealing with the risk of producing an underperforming solution and integrating with the child welfare practice. Like other software-based requests for proposal, the response should include an idea of the milestones, critical path, and contingencies for missing deadlines.

The quality control for predictive analytics projects needs to incorporate metrics for measuring the model performance, in addition to performance of the software or process for building the model. This quality control could define specific thresholds or metrics the model needs to meet. For example, an agency could require a that a model have 95% accuracy; however, such a specific threshold tends to be an extremely difficult, if not impossible, goal for the offeror given the complexities of these problems. Instead, the agency should consider measuring a model’s performance as a tradeoff between several metrics including precision and accuracy.4 For example, a model used to screen and determine the risk of severe abuse or neglect for a child could perform with 80% accuracy but have a false positive rate (e.g., predicting a child is high risk when he/she is actually low risk) of 15%, which would consequently result in a 15% increase of cases in the system. Measuring quality control should also include a plan for evaluating performance over time if the project is integrated into an existing child welfare agency’s business process. Data and policy can change over time which would lead to a change in performance, triggering a need for rebuilding or recalibrating a predictive analytics solution.

4 Vardeman, S. B., & Jobe, J. M. (2016). Basic concepts in metrology and probability modeling of measurement. In Statistical methods for quality assurance: basics, measurement, control, capability, and improvement (pp. 34-39). New York, NY: Springer. doi:10.1007/978-0-387-79106-7.


12

EXAMPLE EVALUATION FACTORS

Here is an example list of evaluation factors for predictive analytics and corresponding sub-factors with sample criteria for measuring them.

Technical Factor:

• Sub-Factor 1: Technical Approach

◦ Criteria 1: The proposal will be evaluated to determine if the offeror demonstrates a reasonable and executable approach to accurately predicting the future case workload for this agency.

• Sub-Factor 2: Staffing Approach

◦ Criteria 1: The staffing plan will be evaluated to determine the offeror’s proposed staff have experience in child welfare and 10+ years of experience in being able to build, validate, and/or maintain predictive analytics models, including published journal articles on similar applications.

• Sub-Factor 3: Corporate Capabilities

◦ Criteria 1: The offeror’s corporate capabilities will be evaluated to determine the depth of the offeror’s abilities to successfully provide predictive analytics solutions including existing solutions within the child welfare field, the ability to develop mitigations for risks, and secure environments for protecting sensitive data.

Past Performance Factor:

• The past performance assessment will assess the confidence in the offeror’s/joint venture member’s ability (which includes, if applicable, the extent of its critical subcontractors’ involvement) to analyze data and build predictive analytics models in order to forecast case workload. The agency will evaluate the offeror’s/the critical subcontractors’ demonstrated record of their ability to meet model performance thresholds on other contracts, stay compliant within the pre-identified timeframe and cost, and produce services in line with agency expectations for integration with existing software. The recency and relevancy of the information, the source of the information, context of the data and general trends in the offeror’s performance will be considered.


13

Summary As more child welfare agencies start to turn to predictive analytics as a tool, developing and issuing RFPs for predictive analytics projects—both research and software design—poses a challenge. Some sections of the RFP are cookie cutter to other RFPs that the agency issues and are therefore very straightforward when issuing an RFP for a predictive analytics project. However, other sections are much more complex and require a detailed understanding of the problem, methods, and ultimate goal of the project. This document has outlined the basic requirements of each of these more complex sections, in addition to highlighting some specific considerations that may be applicable to a predictive analytics project.

Given the different contracting dynamics in every jurisdiction and child welfare agency, as well as the unique requirements of the child welfare problem at hand, it is impossible to issue a blanket fill-in-the-blank RFP that can be used for predictive analytics project. Instead, each child welfare agency interested in pursuing predictive analytics should take this document to its contracting specialists, and work with data scientists and domain experts to ensure that a proposal is prepared that will best ensure the chance of success.


14

Appendix

Standard RFP Sections Nine sections of an RFP are essentially boilerplate from other RFPs that the agency has issued—there is nothing specific to predictive analytics or child welfare that would appear. The following sections will briefly describe the content. While the sections outlined in this document are similar to Federal procurement RFPs, the information contained below are state-centered. Most states have their own state guidelines and regulations and do not adhere to the Federal Acquisition Regulation (FAR)5 required for federal agency RFPs.

Solicitation/Contract

This section often consists solely of a one-page solicitation form. This form gives basic information about the project. Here the offeror will find where and when to submit the bid, contact information for the agency, and the solicitation number. This section also serves as a Table of Contents for the rest of the RFP.

Supplies or Services and Price/Costs

The Supplies or Services and Price/Costs section is a line-by-line list of billable items, also known as Contract Line Item Numbers (CLINs). These line items will include labor, supplies, and other billable items such as travel expenses. This section will also specify pricing structure for the contract.

Packages and Marking

This section details how various line items are to be packed, labeled and shipped or delivered. Sometimes the information that would be in this section is rolled up into Deliveries or Performance or Contract Administration Data.

Inspection and Acceptance

This section aims to protect the agency from poor-quality work and materials. It details the conditions that must be met for the work to be accepted by the agency, as well as how it will be inspected. This may only involve a final examination by the contracting officer at the end of the contract, or it may require an exacting quality assurance process. This section will also include what steps can be taken if the work doesn’t pass inspection.

5 General Services Administration, Department of Defense, National Aeronautics and Space Administration. (2005, March). Federal Acquisition Regulation. Available at: Online link.

https://www.acquisition.gov/sites/default/files/current/far/pdf/FAR.pdf


15

Deliveries or Performance

This section maps out when, where, and how different line items need to be delivered. It could prove helpful to review the Supplies or Services and Prices/Costs while reading this section to easily refer to specific line item numbers. The Deliveries or Performance section is useful to bidders as they develop project plans, as it identifies exactly when goods need to be delivered or services need to be performed.

Attachments and Exhibits

The attachments and exhibits section references the Table of Contents for any appendices and supplementary materials included. Bidders often utilize this section while responding to the Description/Specifications/Statement of Work.

Representations, Certifications, and Other Statements

This section instructs offerors how to demonstrate eligibility to bid on and perform on a contract. It will ask for things including tax information, verification that the offeror’s hiring and employment practices are valid, and small/disadvantaged business status.

Proposal Preparation Instructions

This section enables the contracting officer to quickly read and find essential information in all submitted proposals. These may include font and formatting; organization of the material; policies on amendments, withdrawals, and disclosures; and proposal delivery.

Contract Administrative Data

The amount of contract administration data contained in a specific RFP will vary depending on the type of contract, the agency’s needs, and the agency’s resources, among other things. Some topics this section may cover include status reporting, accounting information, and contact information for key agency personnel. This section informs both you and agency personnel on the ways you will interact during the contract.


16

Acronyms

Acronym Definition ASPE Assistant Secretary for Planning and Evaluation CAMH CMS Alliance to Modernize Healthcare CLIN Contract Line Item Number CMS Centers for Medicare & Medicaid Services COTS Commercial Off-The-Shelf FAR Federal Acquisition Regulation FDA Food and Drug Administration FFRDC Federally Funded Research and Development Center HHS Department of Health and Human Services PII Personally Identifiable Information PWS Performance Work Statement RFP Request for Proposal SOO Statement of Objectives SOW Statement of Work


1

NOTICE

This (software/technical data) was produced for the U. S. Government under Contract Number HHSM-500-2012-00008I, and is subject to Federal Acquisition Regulation Clause 52.227-14, Rights in Data-General.

No other use other than that granted to the U. S. Government, or to those acting on behalf of the U. S. Government under that Clause is authorized without the express written permission of The MITRE Corporation.

For further information, please contact The MITRE Corporation, Contracts Management Office, 7515 Colshire Drive, McLean, VA 22102-7539, (703) 983-6000.

© 2018 The MITRE Corporation.

Documents

Predictive Analytics in Child Welfare - ASPE...Predictive analytics is a set of advanced analytical methods that may enable child welfare agencies to leverage a range of case -level