Research and Evaluation to Strengthen Practice and Demonstrate Outcomes

Elizabeth McFarlane, PhD, MPH

Makati City, Philippines May, 2011

Today’s Talk

•5 Tiered Approach to Evaluation •Prevention Science and Early Childhood •Research to Practice

•Country Reports •Hawaii’s HSP

•Expanding the Knowledge Base •NCS

Five-Tiered Approach to Evaluation 1. Needs Assessment 2. Monitoring and Accountability 3. Quality Review and Program Clarification 4. Achieving Outcomes

• Establish Efficacy, then effectiveness

5. Assessing Impact • Experimental Design to Establish Causality • Replication to Establish Replicability

Program Evaluation

Prevention Science

• IOM call for National Quality Measures for Pediatric Healthcare

• SRCD Social Policy Report 2011 – Global ECD •Focus on equity in ECD programs to include access and quality

Prevention Science

• Early Childhood sets the life course trajectory

• Crucial Domains •Self-regulation: controlling ones emotions, behaviors, attention

•Cognitive Development: language, reasoning, problem solving

•Social Development: capacity to trust, love and resolve conflict

• Parenting mediates most of the impact of poverty

• Investment in ECD saves $$ by preventing later costs

Science Points Toward 3 Types of Services

to Ensure Healthy Development

Basic health care, child care, and early learning opportunities to help all children build and sustain

strong bodies and brains.

Broadly targeted supports for children faced with major risk factors for poor development.

Specialized interventions for children and families with

complex needs.

SPECIALIZED

BROADLY TARGETED

BASIC SERVICES

Hirokazu Yoshikawa, Ph.D (ACF, 2011)

Program Evaluation Research Helps Identify Effectiveness Factors

Not all programs are effective.

Effectiveness factors are key to distinguishing those programs that work from those that do not.

Hirokazu Yoshikawa, Ph.D (ACF, 2011)

Ecologic Setting and Systems Levels and Cross-Cutting Quality Dimensions Britto, Yoshikawa, Boller: Social Policy Report Vol. 25 (2) 2011

Solid Program Infrastructure

•Well-defined model

•Strong leadership

•Clear organizational structure

•Strong hiring and training component

•Supervision and support mechanisms for consultants

•Strategic partnerships

•Community outreach and engagement

•Clear communication

•Evaluation

•Financing 21

Sample of Programs Across Counties/Hawaii’s HSP

Country Program Reports

• Beijing: Uses annual statistics to monitor the success of programs to reduce infant mortality

• Shanghai: Planning a needs assessment of children’s mental health to have data for use in policy and program planning

• Hong Kong: GIS systems to identify injury patterns. Home-based intervention developed to address household risks.

• Hong Kong: Representative Survey of Child and Adolescent Health

• Philippines: Needs assessment of children with autism transitioning to adulthood

• Singapore: Data monitoring to assure impact

Research to Practice Partnership: Hawaii’s Healthy Start Program and JHU

Partnership since 1994

Several Studies: RCT – RWJ, Annie E. Casey, Packard Foundation, MCHB, HDOH RCT – NIMH, NICHD Fidelity Study – CDC EBHV - ACF

Ongoing National Interest in Home Visiting U.S. Congress Patient Protection and Affordable Care Act 2010, provides a $1.5 billion investment in state expansion of evidence-based early home visiting.

Rationale for RCT of Hawaii’s Healthy Start Program

Strong design + program taken to scale

Commitment to evaluation and dissemination

National Interest “In 1990 the US Advisory Board on Child Abuse

described the program as “clearly the star” of US home visitation programs…. A rigorous evaluation is needed … The ideal way of determining program effectiveness would be a randomized controlled trial with multiple outcome measures.”

Institute of Medicine, 1994

Hawaii’s Healthy Start Program

• Goals •To improve family functioning •To promote child health and development •To prevent child abuse and neglect

• Components •Population-based screening •Home visiting

Study Methods

•Sample Cross-sectional study: 6545 births Randomized trial: 684 at-risk families

•Follow-up •Blinded, independent fieldwork staff

•Feedback

•To inform quality improvement

Program Impact

• Decreased partner violence/injury*

• Increased use of nonviolent discipline

• Increased access to a medical home

• Increased continuity of care

• Improved parent-child interaction*

• Improved child development scores*

* At some HSP sites

What We Know from Meta-Analytic Studies of Home Visiting : HV can be effective, but effects are small.

Domain ES Cognitive Development (41 studies) 18* Socio-emotional Development (24) .10* CAN Prevention (7) .32 CAN – Potential Abuse (13) .24* Parenting Stress (4) .21 Parenting Behavior (37) .14* Parenting Attitudes (15) .10* Maternal Education (5) .13* Maternal Employment (7) .02 Public Assistance (3) -.04

ES Key Small .20

Medium .50

Large .80

*p<.05

What We Know from RCTs of Home Visiting

•Programs target the right family outcomes in the right families at the right time.

•Program effects are modest.

•Actual services depart from models.

•Implementation systems weaknesses were the reason for these departure.

•Discrete protocols can improve outcomes.

•Challenges are the same for primary care and home visiting.

Enhancing Hawaii’s Statewide Home Visiting Program to Improve Fidelity and Effectiveness

• Improved Fidelity & Effectiveness

• Cultural Relevancy

• Continuous Quality Improvement System

Evaluation Approach

Conceptual Framework for Implementation Fidelity (Carroll C. et al. (2007)

20

Intervention Fidelity

Coverage Duration & Frequency

Details of Content

Outcomes

Potential Moderators Model complexity & clarity Implementation strategies Participant responsiveness

Evaluation of Implementation Fidelity

Evaluation

Component analysis to identify

“essential” components

Evaluation Approach (Continued)

Community-Based Participatory Research and Quality Improvement Methodologies Used To Improve:

• Home Visiting Infrastructure (e.g. curricula, supervision) • Practice (e.g. family recruitment, visit frequency) • Outcomes (e.g. family functioning, prevention of CAN)

21

The Science to Service Journey…

• What is known is not what is adopted in practice

• Implementation is a process, not an event.

• Adaptation must be evidence-based.

• Without monitoring, fidelity declines over time.

• To achieve intended outcomes, we must: •Adopt efficacious models •Adapt them carefully and cautiously •Achieve fidelity through training and monitoring •Test our results

Expanding the Knowledge Base

The National Children’s Study

• Largest long-term study of children’s health and development ever to be conducted in the U.S.

• Longitudinal study of children, their families, and their environment (before birth through age 21)

• Approximately 100,000 children enables study of important but less common outcomes

* Reappointed 2001 and 2003

Rationale for the National Children’s Study

From The President’s Task Force on Environmental Health and Safety Risks to Children, 2000*

• Compared to adults, children are especially vulnerable to environmental exposures – metabolism, behavior

• Exposures to some agents demonstrate potential for serious developmental effects – lead, prenatal alcohol

• Current known exposures of high frequency – pesticides, violence, media

• Numerous high burden conditions with suspected environmental contribution – learning disabilities, autism, diabetes, asthma, birth defects, premature birth

• Existing research too limited in size and scope to answer the questions

• Life-course (longitudinal) design needed to correctly link with multiple exposures and multiple outcomes

Converging factors

• Numerous experiences of adverse effects of environmental exposures on children’s health and development

• Many identified environmental exposures with concerns about toxic and other adverse effects in children

• Diseases and developmental conditions with possible environmental causes or contributions

Priority Environmental Exposures

• Physical environment: housing, neighborhoods and communities, climate, radiation…

• Chemical exposures: air, water, soil, food, dust, industrial products, pharmaceuticals… • complex ubiquitous low-level exposures • unique exposures (special sub-studies)

• Biological environment: intrauterine, infection, nutrition; inflammatory and metabolic response…

• Genetics: genetic components of disease; effects of environmental exposures on gene expression…

• Psychosocial milieu: influence of family, socio-economics, community, stress…

Examples of Hypothesis-defining questions

• How is asthma incidence and severity influenced by the interaction of early life infection and air quality?

• Are assisted reproductive technologies (ART) at increased risk of fetal growth restriction, birth defects, and developmental disabilities?

• Does impaired maternal glucose metabolism during pregnancy cause obesity in children?

• How does high level exposure to media content in infancy affect development and behavior in children?

• Does pre-and post-natal exposure to endocrine-active environmental agents alter age at onset, duration, and completion of puberty?

Priority Health Exposures, Outcomes

Priority Exposures

Examples

Physical Environment Housing quality, neighborhood

Chemical Exposures Pesticides, phthalates, heavy metals

Biologic Environment Infectious agents, endotoxins, diet

Genetics Interaction between genes and environment

Psychosocial milieu

Family structure, socio-economic status, parenting style, social networks, exposure to media and violence

Priority Health Outcomes

Examples

Pregnancy Outcomes Preterm, Birth defects

Neurodevelopment & Behavior

Autism, learning disabilities, schizophrenia, conduct and behavior problems

Injury Head trauma, Injuries requiring hospitalizations

Asthma Asthma incidence and exacerbation

Obesity & Physical Development

Obesity, diabetes, altered puberty

National Children’s Study Sample

105 Locations

Selection of neighborhoods

All or a sample of households within neighborhoods

All eligible women in the household

~4 million births in 3,141 counties

All Births in the Nation

Sample of Study Locations

Sample of Study Segments

Study Households

Study Women

Study Participation

• Recruit from pre-conception/early pregnancy • Diverse populations: ethnicity, SES, family structure • Minimum of 16 in-person home and clinic-based visits

(pre-conception through 21) • Contact by telephone, computer, mail-in questionnaires

every 3 months until 5 years old; then annually • Environmental (air, water, soil, dust) samples from

child’s environment (home, school, day care) • Psychosocial, demographic, neurodevelopmental,

neighborhood and contextual data • Biological samples from mother, father, child

What will the NCS Mean to our Children’s Well-Being?

• Identification of environmental factors which cause or contribute to health, development and behavior problems • Asthma, injury, obesity, autism, ADHD, prematurity…

• Understanding the biology and genetics of health, development and behavior

• Evidence-based information on which to base decisions about practice and policy regarding children’s physical and mental health

• Economic benefits: cost avoidance

• Resource for future research

So, the NCS will Provide…

• The answer to concerns about known exposures during childhood to potential toxicants

• The power to determine absence of effects or benefit of exposures to various products important for our society

• Causal factors for a number of diseases and conditions of children with suspected environmental causes

• How multiple causes interact to result in multiple outcomes

• Large sample size required to apply knowledge of the human genome to understand multi-factorial genetic conditions

• Identification of early life factors that contribute to many adult conditions

• A national resource to answer future questions by using stored biological and environmental samples and the extensive data for decades to come

Thank You! Maraming Salamat!

Questions?