1April 14, 2010 1

Teenage Pregnancy Prevention: Replication

of Evidence-based Programs (Tier 1)

U.S. Department of Health and Human Services

Office of Adolescent HealthApril 14, 2010

3:00-5:00pm ET

2April 14, 2010 2

Purpose of Today’s Call Review the Teenage Pregnancy Prevention:

Replication of Evidence-Based Programs Funding Opportunity Announcement, including the purpose, eligibility requirements, and how to apply for funds

Will not be answering questions specific to individual applications

3April 14, 2010 3

IntroductionsOffice of Adolescent Health Evelyn Kappeler, Acting Director Alice Bettencourt, Acting Deputy Director Allison Roper, Public Health Analyst Amy Margolis, Public Health Analyst Jennifer Gannon, Program Specialist Miryam Gerdine, Public Health Analyst

Office of Grants Management Karen Campbell, Director

Overview of the Office of Adolescent

Health & Teenage Pregnancy Prevention Initiative

5April 14, 2010 5

Office of Adolescent Health (OAH)

Consolidated Appropriations Act, 2010 directed that a new OAH be established

Responsible for implementing and administering new grant program to support evidence-based teen pregnancy prevention approaches

Coordinates adolescent health programs and initiatives across the U.S. Department of Health and Human Services

6

Office of Grants Management

Official signatory for obligating federal grant funds

Official signatory for all grant business

Monitor all business/financial transactions on grants for compliance to Federal Regulations (including interpretation of Federal Regulations)

Kathleen SebeliusSecretary

Office of Public Health and Science

ASSISTANT SECRETARY FOR HEALTHHoward Koh, MD, MPH

Principal Deputy AssistantSecretary for Health

Wanda Jones, DrPH

Senior Advisor to the ASHRosemarie Henson, MPH, MSW

Regional HealthAdministratorsRegions I-X

National Vaccine Program Office

Deputy Assistant SecretaryBruce Gellin, MD, MPH

Deputy AssistantSecretary for Health(Healthcare Quality)

Don Wright, MD, MPH

Office of the Surgeon GeneralSurgeon General

Regina Benjamin, MD, MBAVADM, USPHS

Office of Disease Prevention and Health Promotion

Deputy Assistant SecretaryPenelope Slade-Sawyer, PT, MSW

RADM, USPHS

Office of Population AffairsDirector

Susan Moskosky(Acting)

Office on Women’s HealthDirector

Frances Ashe-Goins, RN, MPH(Acting)

President’s Council on Physical Fitness & Sports

Executive DirectorShellie Y. Pfohl, MS

Office of Research IntegrityDirector

Don Wright, MD, MPH(Acting)

Office of HIV/AIDS PolicyDirector

Christopher Bates, MPA

Office of Minority HealthDeputy Assistant Secretary

Garth Graham, MD, MPHLCDR, USPHS

Office of Commissioned Corps Force Management

DirectorDenise Canton, JD, RN

RADM, USPHS

Deputy AssistantSecretary for Health

(Science and Medicine)Anand Parekh, MD,

MPH

Office of CommunicationsDori Salcido

Office of Adolescent HealthDirector

Evelyn Kappeler(Acting)

Office for HumanResearch Protections

Director Jerry Menikoff, MD, JD

8April 14, 2010 8

LocationU.S. Department of Health

& Human Services

Office of the Secretary of DHHS

Office of Public Health and Science

Office of Adolescent Health

Office of Grants Management

9April 14, 2010 9

Teenage Pregnancy Prevention Initiative Consolidated Appropriations Act, 2010 (Public

Law 111-117)- $110 million

$75 million - replicate program models proven effective through rigorous evaluation (Tier 1)

$25 million - research and demonstration grants to develop, replicate, refine, and test additional models and innovative strategies (Tier 2)

$10 million - training and technical assistance, evaluation, outreach, and additional program support activities

Introduction to TPP Tier 1:

Replication of Evidence-based

Programs Funding Announcement

11April 14, 2010 11

Purpose of Tier 1 Funding Announcement

To support the replication of evidence-based program models that are medically accurate, age appropriate, and have proven through rigorous evaluation to reduce teenage pregnancy, behavioral risks underlying teenage pregnancy, or other associated risk factors.

12April 14, 2010 12

Target Populations Individuals 19 years

of age or under at program entry

Applicants should clearly define target populations by age groups or priority populations within a defined geographic area with high teen birth rates.

13April 14, 2010 13

Who’s eligible to apply? “Funds made available … shall be for making

competitive contracts and grants to public and private entities” (Consolidated Appropriations Bill, 2010) Nonprofit organizations For-profit organizations Universities and colleges Research institutions Hospitals Community-based organizations Faith-based organizations Federally recognized or state-recognized

tribal governments State and local governments State and local school districts Political subdivisions of States

14April 14, 2010 14

Funding Ranges for Tier 1

Range A - $400,000 to $600,000 per yr Range B - $600,000 to $1,000,000 per yr Range C - $1,000,000 to $1,500,000 per yr Range D - $1,500,000 to $4,000,000 per yr

* Applicants may only apply for one funding range under this announcement.

15April 14, 2010 15

Cost-Sharing or Matching Neither cost-sharing nor matching are

required

Applicants are encouraged to include participation by stakeholders in the community as an indicator of community support for the project

An indication of institutional support from the applicant and its collaborators indicates a greater potential of success and sustainability of the project

16April 14, 2010 16

Award Information Type of Award: Cooperative Agreements

Number of awards: Up to 150

Project Period: Up to 5 years

Funding Range: $400,000 - $4,000,000

Start Date: no later than September 30, 2010

Applicants may only submit one application for consideration

17April 14, 2010 17

Any Questions?

Identification of Evidence-based

Programs Eligible for Replication

19April 14, 2010 19

Identification of Evidence-based Programs

Independent, systematic review of evidence base conducted by Mathematica Policy Research, under contract to HHS

Steps of the Review Find potentially relevant studies Screen studies to review Assess the quality of studies Assess the evidence of effectiveness

More detailed information available at http://www.hhs.gov/ophs/oah

20April 14, 2010 20

Finding Relevant Studies Review of reference lists from earlier research

syntheses

Search of relevant research and policy organizations’ websites

Public call for studies to solicit new and unpublished research

Keyword search of electronic databases

21April 14, 2010 21

Screening Criteria Use quantitative data and statistical analysis

and hypothesis testing to measure impacts

Measure impact on at least one sexual risk behavior or its health consequences: sexual activity (initiation, frequency, number of partners), contraceptive use, sexually transmitted infections, pregnancies, or births

Focus on youth ages 19 or younger in the United States at start of program

Been conducted or published since 1989

22April 14, 2010 22

Assessing Study Quality High, Moderate, or Low Rating based on:

- Study design - Reassignment- Attrition - Confounding factors- Baseline equivalence

High rating = random assignment studies with low attrition and no sample reassignment

Moderate rating = quasi-experimental designs with well-matched comparison groups at baseline; certain random assignments that didn’t meet all criteria for high rating

23April 14, 2010 23

Assessing Evidence of Effectiveness

Supported by at least one high- or moderate-rated study showing a positive, statistically significant impact on at least one priority outcome (delay in sexual activity; increase in contraceptive use; decrease in STIs, pregnancies, or births) for either the full study sample or a key subgroup (defined by gender or baseline sexual experience)

24April 14, 2010 24

Program Models Eligible for Replication 28 Evidence-based program models currently

identified as eligible for replication with Tier 1 Funding

http://www.hhs.gov/ophs/oah/prevention/research/programs/index.html

Appendix A of Funding Announcement

All studies that were reviewed but didn’t make the Tier 1 list and the rationale for why the study didn’t meet the review criteria are available in a searchable database at www.hhs.gov/ophs/oah

Program Models Currently Eligible for Replication

Aban Aya Youth Project Adult Identity Mentoring (Project

AIM) All4You! Assisting in Rehabilitating Kids

(ARK) Be Proud! Be Responsible! Be Proud! Be Responsible! Be

Protective! Becoming a Responsible Teen

(BART) Children’s Aid Society (CAS) –

Carrera Program Comprehensive Abstinence and

Safer Sex Intervention ¡Cuídate! Draw the Line/Respect the Line FOCUS

HIV Risk Reduction Among Detained Adolescents

Horizons It’s Your Game: Keep it Real Making a Difference! Making Proud Choices! Project TALC Promoting Health Among Teens! Reducing the Risk Rikers Health Advocacy

Program (RHAP) Safer Sex Seattle Social Development

Project SiHLE Sisters Saving Sisters Teen Health Project Teen Outreach Program What Could You Do?

26

Where to Learn More About the 28 Program Models

Intervention Implementation Reports on OAH website (www.hhs.gov/ophs/oah) Intervention Name and Developer Program Description Target Population Curriculum Materials Training and Technical Assistance Research Evidence

27April 14, 2010 27

Can an organization apply to replicate program models not

in Appendix A?Yes, but only if all of the following criteria are met:

1. Research or evaluation of program model was not previously reviewed (previously reviewed evidence will not be re-reviewed)

2. Research or evaluation meet the screening and evidence criteria

3. Application must include all relevant research and evaluation (not included in 100 page limit)

4. Application must be received by May 17, 2010

28April 14, 2010 28

Any Questions?

Program Expectations

30April 14, 2010 30

Overview of Program Expectations

Implement an evidence-based program model Maintain fidelity to the program model Address the target population Ensure medical accuracy and age-

appropriateness Engage in phased-in implementation period Collect and report performance measurement

data Adhere to evaluation expectations

31April 14, 2010 31

Maintaining Fidelity to Program Model

Must maintain fidelity to the “core components” of the original evidence-based model that led to the outcomes associated with the program

“Core Components” are those parts of the curriculum or its implementation determined by the developer to be the key ingredients related to achieving the outcomes associated with program

32April 14, 2010 32

Can the Program Models be Adapted?

Minimal adaptations are allowed: Changing names or details in a role play Updating out-dated statistics Adjusting reading and comprehension levels Making activities more interactive

May propose adaptations to make program more relevant to ethnic, racial, or linguistic characteristics of the population to be served as long as core elements aren’t affected

Significant adaptations (adding activities, changing sequence of activities, replacing supplementary materials) are not allowed under Tier 1

33April 14, 2010 33

Demonstrating Ability to Replicate Program Models

with Fidelity Demonstrate effectiveness of replication

strategy

Ensure facilitators delivering program have been or will be formally trained

Receive training on acceptable adaptations or propose adaptations for approval

Monitor and document program implementation to ensure fidelity

Provide MOUs stating that all partners have agreed to implement with fidelity

34April 14, 2010 34

Medical Accuracy & Age-Appropriateness

Funded programs will need to ensure that information provided is age appropriate, and scientifically and medically accurate

Materials will be reviewed by OAH prior to implementation to ensure medical accuracy

Full curricula should NOT be submitted with the application. Program materials will be submitted to OAH for review and approval during the phased-in implementation period of the first grant year.

35April 14, 2010

35

Phased-In Implementation Period

Funded recipients will engage in a planning, piloting, and readiness period for the first 6 to 12 months of funding

During this period, grantees will: Continue to assess needs and resources Finalize goals, objectives, and logic model Assess program fit Build organizational capacity Finalize implementation plans Pilot test program

The length of the phased-in implementation may vary by grantee depending on implementation readiness

OAH approval is required before full-scale implementation

36April 14, 2010 36

Evaluation Strategies Monitoring and reporting on program

implementation and outcomes through performance measures for all grantees

Grantee-level evaluations for projects funded in Ranges C and D

Federal evaluation of a selected subgroup of all grantees

37April 14, 2010 37

Performance Measures Developed by OAH during first year of program All grantees will be expected to collect and report

on common set of performance measures to assess program implementation and outcomes

Training and technical assistance will be provided by OAH

Anticipated categories for measures: Output measures Fidelity/adaptation Implementation and capacity building Outcome measures Community data

38April 14, 2010 38

Evaluation for Funding Ranges A & B

Must be able to demonstrate ability to collect and report on common set of performance measures to assess: program implementation and outcomes

Not expected to conduct rigorous independent grantee-level evaluation

39April 14, 2010 39

Evaluation for Funding Ranges C & D

Rigorous independent grantee-level evaluation design unique to proposed project Use either random assignment or quasi-experimental

design OAH will review and assess proposed evaluation designs OAH approval required before implementing evaluation

plan OAH will provide training & TA on evaluation – general &

project specific.

See Appendix C in the FOA for detailed guidance

Budget 20-25% to support evaluation activities, but not more than $500,000 per year

40April 14, 2010 40

Federal-level Evaluation of TPP

Grantees from all funding ranges are required to participate in Federal evaluation, if selected Will agree to follow all evaluation protocols

established by HHS Will no longer be expected to have separate

grantee-level evaluation and will be required to redirect evaluation budget to support activities related to the Federal-level evaluation

Decisions of which grantees will participate in the Federal evaluation will be made by end of the 1st grant year

41

Items Requiring OAH Approval

OAH approval required for: Medical accuracy of curricula and program

materials Proposed adaptations Evaluation plans Full-scale implementation

Full curricula should NOT be submitted with the application. Program materials will be submitted to OAH for review and approval during the phased-in implementation period of the first grant year.

April 14, 2010

42April 14, 2010 42

Any Questions?

Application Contents

44April 14, 2010

44

Contents of Application Submission

Abstract (one-page)

Project Narrative (no more than 50 pages) Organizational Capability Statement Project Management Need Statement Model to be Replicated and Project Approach Target Population Program Goal(s), Objectives, and Activities Work plan and Timetable Collaborations and Description of MOUs Performance Measurement Evaluation

Appendices

Budget Narrative/Justification

No more than

100 pgs

45April 14, 2010 45

Project Abstract (1 page) Clear, concise description of the project that

can be understood without reference to other parts of the application. Should include: Project title Applicant contact information Type of organization applying Overarching goal(s) Evidence-based program model to be replicated Geographic area to be served Target population

46April 14, 2010 46

Project Narrative Formatting

Double-spaced 8 ½ x 11 inch (letter-size) pages 1-inch or larger margins on top, bottom, and

both sides At least 12 point font All pages, charts, figures, and tables should

be numbered

47April 14, 2010 47

Project Narrative:Organizational Capability

Current capability to organize and operate effectively and efficiently

Decision-making authority and structure Organization’s experience, expertise and previous

accomplishments in the area of teen pregnancy prevention

Previous partnerships and strategies used to address teen pregnancy prevention

How various sites and outside resources will be managed logistically and programmatically

48April 14, 2010 48

Project Narrative:Project Management

Plans to govern and manage the execution of the overall program

Governance structure, roles/responsibilities, operating procedures, composition of committees, workgroups, terms and associated leaders, and communications plans

How plans and decisions are developed and documented and issues/risks managed

Specify mechanisms to ensure accountability among community participants and incremental progress in achieving milestones

49April 14, 2010 49

Project Narrative:Need Statement

Geographic area to be served Benefit for the target population Incidence of teen births in the area Documentation of sexually transmitted disease

rates, socio-economic conditions including income levels, existing services and unmet needs in the service area

Unique challenges and barriers facing proposed population

50April 14, 2010 50

Project Narrative:Model to be Replicated

Rationale for choosing program model for replication and how approach is based on previous practice and community needs assessment

Lessons learned from previous projects of this type Implementation site(s) Plans to implement model with fidelity Proposed adaptations to program model Plans to train staff and obtain implementation

materials Plans to coordinate, integrate, and link to existing

services within the service area Program management plan

51April 14, 2010 51

Project Narrative:Target Population

Describe target population using statistical data and other community factors

Provide realistic estimates of overall number of program participants and number participating in proposed project site(s)

Describe expected number of participants during first and second year of implementation; break out by age, race, and ethnicity

Describe age appropriateness of program model

52April 14, 2010 52

Project Narrative: Program Goal(s), Objectives, and

Activities Specific program goal(s) and up to 6 outcome

objectives

SMART objectives S – Specific M – Measurable A – Achievable R – Realistic T – Time-framed

Logic Model (5-year) – visual representation of relationships between proposed resources, planned activities, and desired outputs and outcomes

53April 14, 2010 53

Project Narrative:Work plan & Timetable

Detailed five-year work plan Concise, easy-to-read overview of goals, strategies,

objectives, measures, activities, timeline, and those responsible

Include each activity associated with the replication, the proposed time frame for the activity, and the responsible staff

Timetable for first year of project

54April 14, 2010 54

Project Narrative:Collaborations

Detail intent to coordinate with other community agencies and not duplicate existing efforts

Describe expertise and capabilities of other partnering agencies

Identify community stakeholders

Include MOUs from each participating site, stakeholders, and outside resources (if applicable) in the Appendices

Detail the specific nature of involvement and level of commitment of each stakeholder

55April 14, 2010 55

Project Narrative:Performance Measurement

Monitor progress on uniform set of process and outcome performance measures

Describe capability to implement monitoring and reporting systems to aid in internal data collection around metrics for successful achievement of performance measures

56April 14, 2010 56

Project Narrative:Evaluation (Ranges A & B)

Applicants in Funding Ranges A & B: Describe capacity to collect and report on

common set of performance measures to assess: program implementation and outcomes

Describe how data will be used for ongoing program improvements

57April 14, 2010 57

Project Narrative:Evaluation (Ranges C & D)

Applicants in Funding Ranges C & D only: Provide clear and fully developed grantee-level,

independent evaluation plan in accordance with criteria in Appendix C of the FOA

Describe proposed project, experimental design, processes to be tested, theory upon which intervention is based, proposed questions/hypotheses, data collection instruments, sampling and data collection plan, and data analysis plan

Describe how data will be used for ongoing program improvements

Describe plans to ensure confidentiality of data Include a MOU and CV from the independent evaluator

58April 14, 2010 58

Project Narrative:Federal Evaluation

Successful applicants must agree, if selected, to participate in a Federal evaluation, conducted by an independent contractor through a separate competitive award process

5959

Appendices Resumes for Project Director and detailed position descriptions Program logic model (see the Application Kit for example) Memoranda of Understanding from all participating sites Memorandum of Understanding with independent evaluator Curriculum Vitae of independent evaluator Memoranda of Understanding from all partners Organizational chart Applicant organization’s Federal-Wide Assurance Proof of nonprofit status, if applicable For applicants seeking to replicate a program model not on

the list in Appendix A, all materials that support the claim that the model meets evidence-related criteria (not counted in 100 page limit)

60April 14, 2010 60

Budget Narrative/Justification

State the funding range requested

Outline proposed costs that support all project activities

Thoroughly describe how the proposed costs are derived Breakdown each line item and provide an

explanation of the costs Personnel should include salary per person and

percent of time each person will spend on grant

Specify the source of in-kind contributions (if applicable), and how the valuation of that contribution was determined

61April 14, 2010 61

Budget Information: Explanation of Line Items for

424aa. Personnelb. Fringe Benefits- based on bylaws of

organization; taxes; social security; insurance benefits; percent varies by organization

c. Travel- include OAH annual meeting & 3 regional trainings

d. Equipment- items over $5,000 per unite. Supplies- items under $5,000 per unitf. Contractual- may include independent evaluatorg. Construction – not allowed h. Other

62

Funding Restrictions Funds may not be used:

For building alterations or renovations, construction, fundraising activities, political education or lobbying

To supplant or replace current public or private funding

To supplant on-going or usual activities of any organization involved in the project

To purchase or improve land, or to purchase, construct, or make permanent improvements to any building

To reimburse pre-award costsApril 14, 2010 62

63April 14, 2010 63

Other Required Application Forms

Application for Federal Assistance (SF 424) Budget Information for Non-Construction

Programs (SF-424a) Project Abstract Project Narrative Attachment Form Budget Narrative Attachment Form Assurances for Non-Construction Programs (SF-

424B) Disclosure of Lobbying Activities (SF-LLL) HHS Certifications (08-2007)

64April 14, 2010 64

Any Questions?

Application Submission Instructions

66April 14, 2010 66

Receipt Deadlines Letter of Intent – May 3, 2010

Include descriptive title, funding range, and contact information

Submit via fax to (240) 453-2801 or via email to oah.gov@hhs.gov

Tier 1 Applications Electronic submission – June 1, 2010 by 11:00pm ET Paper submission – June 1, 2010 by 5:00pm ET Applications to replicate a program not on the list in

Appendix A – May 17, 2010 by 5:00pm ET for paper applications or 11:00pm ET for electronic applications (must include all relevant new evidence with the application – evidence will not be counted in 100 page limit)

67April 14, 2010 67

Electronic Submissions Grants.gov- http://www.grants.gov/

GrantSolutions- https://www.grantsolutions.gov/

Register well in advance

Must be submitted no later than 11:00pm Eastern Time on June 1,2010

All required hardcopy original signatures and mail-in items must be received by the Office of Grants Management Grant Application Center no later than 5:00pm Eastern Time on June 2, 2010

Applications are not considered valid until all electronic components, hardcopy original signatures, and mail-in items are received

68April 14, 2010 68

Paper Submissions Must be received no later than 5:00pm Eastern Time

on June 1,2010 Address all materials to:

Office of Grants ManagementOffice of Public Health and Science (OPHS)Department of Health and Human Services (DHHS) c/o Grant Application Center1515 Wilson Blvd., Suite 100Arlington, VA 22209 Attention: Office of Adolescent Health

Include the following on the lower left corner of your mailing envelope: “ATTENTION: OFFICE OF ADOLESCENT HEALTH – TIER 1 (NEW GRANT APPLICATION)”

69April 14, 2010 69

Application Tracking Form Applicants are strongly encouraged to

complete and submit this form at the time of application submission Fax to (240) 453-2801, or Email to oah.gov@hhs.gov

Used by the OAH to track grant applications

Form available in Appendix D of FOA or on the OAH website under “Current Grant Announcements” - “Application Kit”

70April 14, 2010 70

Application Tips If submitting a hardcopy by mail or hand

delivering… Do NOT staple or otherwise bind your application Do NOT send any extraneous materials such as videos,

books, etc. Do NOT send double-sided information/ pamphlets Do NOT submit the proposed curriculum or educational

materials- only include the Table of Contents in the application

DO double space on 8 ½ X 11 paper DO submit an original and two copies

71April 14, 2010 71

Application Tips Read the entire FOA and application kit BEFORE

writing Write the One Page Summary - AFTER the entire

narrative is complete Read the review criteria Read the application instructions Do NOT exceed the 50 page limit for the project

narrative OR the total page limit of 100 pages for the full application

Turn in application no later than 5pm ET (hard copies) or 11pm (electronic copies) on June 1, 2010 Allow time for any unforeseen difficulties with the

on-line application process, etc.

72April 14, 2010 72

Application Tips FOA is the primary guide to programmatic

requirements

Follow what the FOA says

Review & Selection Criteria

74April 14, 2010 74

The Application Review Process: Review Criteria

Project Approach and Work Plan (35 points) Organizational Capacity and Experience (25

points) Project Management and Staffing (15 points) Evaluation Plans and Performance

Measurement (15 points) Budget (10 points)

75

Selection Process Objective Review Committee

Expert peer reviewers Federal Staff

Applications reviewed according to the published review criteria

76

Final Award Decision Criteria

Score and rank order given by the Objective Review Committee

Availability of funds Representation of programs across communities,

including varied types of interventions and evidence-based strategies

Geographic distribution of projects Inclusion of communities of varying sizes Feasibility of evaluation plan Inclusion of range of populations

disproportionately affected by teenage pregnancy

77

Notification of Funding Notice of Grant Award notifies successful

applicant of selection

Includes any conditions on the award

78April 14, 2010 78

Any Questions?

79April 14, 2010 79

Thank you! Office of Adolescent Health

Website - www.hhs.gov/ophs/oah Email – oah.gov@hhs.gov Phone – (240) 453-2806

Office of Grants Management Email – karen.campbell@hhs.gov Phone – (240) 453-8822