The Family Nurse Partnership (FNP) Programme

Developing the Evaluation Framework

FNP Programme

• It is an intensive nurse-led home visiting programme that enables the family nurse to visit the same client from early pregnancy until the child is two

• It is an intervention for young, first time mothers, who meet the broad eligibility criteria (19 and under at LMP, keeping their baby, living within geographical boundaries, <28 wks gestation at recruitment)

• Programme aims:Improve pregnancy outcomes Improve child developmentImprove the economic self-sufficiency of the family

FNP in UK• England since April 2007. Implemented in 55 LA/ PCT areas.• FNP is now being tested in Scotland for the first time in NHS

Lothian City of Edinburgh Community Health Partnership (Edinburgh CHP). First babies enrolled in March 2010.

• 2 teams in NHS Tayside, covering Dundee, Angus and Perth and Kinross CHP’s.

• England: Formative evaluation of the first ten sites reported 2011

• England: RCT in 18 sites which will report, initially, in 2013. The RCT will assess what the benefits and costs associated with FNP, looking closely at prenatal health behaviours, and early child health outcomes.

FNP EvaluationEvaluation Tem: Scottish Centre for Social Research

The overall aim: evaluate the implementation of the programme in Scotland (Lothian)

Specific questions:• Is the programme being implemented as intended? If not, why not?• How does the programme work in Scotland (Lothian)? • Wider implications for implementing the programme in Scotland.

Evaluation implementation: Monitoring and Evaluation Framework: Internal (FNP data) and External (stakeholder interviews, qualitative panel and focus groups) relevant to outcomes of interest in Scotland

How we developed M+E framework

• Series of focused meetings facilitated to achieve consensus on the what the programme was intending to achieve (outcomes) and how this was intended to operate (processes and assumptions)

• Produced two logic models: a Google Earth view and an implementation model

• These provided framework for the M+E

Assesstransferability

and effectivenessin Scotland

Dev't & maintenanceof a skilled FN team

Improvepregnancy and birthoutcomes

Home visits antentally

as per FNPschedule

Internal monitoringExternal evaluation

trainingsupervision

team meetingslearning sets

Improved nursing skills/practice e.g. in promoting attachment

self-efficacy& taking an ecological/PH approach

Enhanced understanding of FNP delivery

and effectiveness

Improvechild health

and dev't

Home visits (postnatally)

until child is 2 yrsas per FNPschedule

Improveparental life course

Improved child health & development

reduced A+E visits and hosp'n for injuries

redc'n in cases of neglectimproved HOME scores

responsivered'n in delays in language

+ cognitive devt

Better maternal healthReduced substance use

Improved maternal mental healthReduced PND

More health enhancing behaviours (diet, physical activity etc)

Parents engage in child health-enhancing behaviours

% of immunisations from 0-2 yearsbreastfeeding: initiation & maintenance

italics = outcomes listed in service level agreement

Enhanced infrastructure(in Edinburgh?)

to support vulnerable mothers

Reduced neonatal risk factorse.g. preterm deliveries, birthweight,

neurodevelopmental impairment,foetal alcohol syndrome

Improved adolescent outcomesred'n in child abuse/neglect (0-15)

fewer arrests + adjudication for bad conduct

Early childhoodfewer safety hazards in home

more stimulating home environreduced recorded unintentional injuries

improved school readiness/ pre-school language

fewer child beh probs within clinical range

Improved health behaviours in pregnancyReduced substance use (tobacco, alcohol and drug)

More health enhancing behaviours (diet, physical activity etc)

Improved maternal mental healthAppropriate uptake of preventative services

Parents demonstrate more competent care & improved parenting

e.g.good attachmentstimulating care and environs

safety in homeuse of other progs/community resources

involvement of Dads

Enhanced econ self sufficiencyno of months mums are working

Parental lifecourse (3-4 yrs after prog)fewer pregnancies

more space between 1st and 2nd pregnanciesno of months in workforce

less federal assistance/food stampshigher rates of living with father of child

higher rates of marriage

Enhanced parental life course (13 yrs after prog)

Less reliance on benefitsFewer arrests /

convictions/days in jailInc space between 1st & 2nd C'n

SHORT TERM OUTCOMES

LONG TERM OUTCOMES

[based on US trials]

INTERMEDIATE OUTCOMES[from US trials: 4-6 yr follow up]

PROGRAMME GOALS

ACTIVITIES

Preliminary logic model Preliminary logic model using EB and incorporating outcomes in service level agreementusing EB and incorporating outcomes in service level agreement

Improvepregnancy and birthoutcomes

Home visits antentally

as per FNPschedule

Improveparental

life course

Improvechild health

and dev't

Home visits (postnatally)

until child is 2 yrsas per FNPschedule

Reduced neonatal risk factorse.g. preterm deliveries, birthweight,

neurodevelopmental impairment,foetal alcohol syndrome

Dev't & maintenanceof a skilled FN team

Assesstransferability

and effectivenessin Scotland

trainingsupervision

team meetingslearning sets

Improved child health & development

reduced A+E visits and hosp'n for injuries

redc'n in cases of neglectimproved HOME scores

responsivered'n in delays in language

+ cognitive devt

Better maternal healthReduced substance use

Improved maternal mental healthMore health enhancing behaviours

(diet, physical activity etc)

Internal monitoringExternal evaluation

Improved nursing skills/practice e.g. in promoting attachment

self-efficacy& taking an ecological/PH approach

Enhanced understanding of FNP delivery

and effectiveness

Contribute toa Healthier,

Wealhier....Fairer Scotland

Contribute toRelevant National Outcomes ....

Cn have best start in lifeImproved life chances for children,

yp and familes at riskYp are successful learners

Better employment opportunitiesLonger, healthier lives

Tackled significant inequalitiesLive lives free from crime and danger

Public services are high quality, continually improving

& responsive to people's needs

....and most relevant HEAT TargetsH2: dental registrationsH3: healthy child weight

H4: Alcohol screeing and BIH6 Smoking cessation

H7: breastfeeding

Contribute to Edinburgh outcomes (as per SOA)

Edinburgh's children are healthyMental health and wellbeing is improved

Our children have the best startin life, are able to make + sustain relnships

and are ready to succeedChildren's early years' devt, learingand care experiences are improved

so that they are ready for school

black italics = outcomes listed in service level agreementBlue text = link to Scottish outcomes and/or programmes

Improved adolescent outcomesUS evidence indicates:

e.g. red'n in abuse/neglect reduced antisocial behaviour/crime

fewer arrests + adjudication for bad conduct

In additition, we anticipate:improved educational attainment

Cycle of deprivation interruptedOffspring themselves

have better parenting skills

Money saved by the state

Enhanced parental life course (13 yrs after prog)

Less reliance on benefitsFewer arrests /

convictions/days in jailBigger interval between

1st & 2nd C'n

Parental lifecourse US evidence indicates (3-4 years after prog):

fewer pregnanciesmore space between 1st and 2nd pregnancies

no of months in workforceless state assistance

higher rates of living with father of childhigher rates of marriage

In additition, we anticipate:better maternal /paternal mental health

Less domestic abuseBetter use of services

More accessing good quality child care

influence/inform thinking on the role/delivery of

community nursing

Enhanced infrastructurein Edinburghto support

vulnerable mothers

LONG TERM OUTCOMES and IMPACT 5+ years after end of intervention

INTERMEDIATE OUTCOMESup to 5 years after end of intervention

SHORT TERM OUTCOMESACTIVITIESPROGGOALS

Parents engage in child health-enhancing behaviours

% of immunisations from 0-2 yearse.g. breastfeeding: [H7]

Registration with dentist? [H2]Better weaning practices/diet [link to H3]

Uptake of Healthy StartImproved use of community services

Improved health behaviours in pregnancyReduced substance use (tobacco, alcohol and drug)

Better diet, more PAImproved maternal mental health

Appropriate uptake of preventative servicesUptake of screening services (CEL 31)

Use of Vit D supplements & folic acid (CEL 36)

Parents demonstrate more competent care & improved parenting

e.g.good attachmentstimulating care and environs

safety in homeuse of other progs/comm'ty resources

involvement of Dads

Enhanced econ self sufficiency

Early childhoodUS evidence indicates:

fewer safety hazards in homemore stimulating home environ

reduced recorded unintentional injuriesimproved school readiness/

pre-school languagefewer child beh probs

In addition, we antcipate:Less dental disease at P1

Inc fluoride varnish applicationsLower BMI

Better mental healthImproved relationship

between child and mother

FNP: High level strategic model

supportive local (HB, LA, CHP)and national (SG)

infrastructure

Office and ITsystems & staffto manage data/

monitor FNP

Recruitment & capacity building of team

eg. training,supervision, learning sets

awareness raising /liaison with agencies/services

incl health care providers

prod'n of reports of monitoring data(specify timing/frequency)

Skilled team

Better pregnancy outcomes

Improvedchild health

+ devt

Improved parental

life course

processing/maintaining

info andsystems

ACTIVITIESINPUTS SHORT TERM OUTCOMESOUTPUTS

Programme budgetcirca £1.6m over 3 years

all requisite trainingattended by FN team [1]

weekly supervisionof FNs [2]

team of 6 FNs,supervisor & support team

with FNP competencies [3]

145 women enrolledwho meet eligibility

criteria [8]

FNs have caseload of

max 25 families [9]clients receive at least 80% of weekly visits for 1st month

following enrolmentthen alternate weeks until birth [11]

clients receive at least 65%of weekly home vists

for first 6 weeks post partum [12]

Clients receiveat least 65% of home vistis on alternate weeks

from age 6 weeks to 21 months [13]

monthly home vists between 21months

and 2 years [14]Implementation records

and all assessment forms completed [R1]

all monitoring data entered onto ITsystems [R2]

Client understands how & demonstrates

sensitive/competentcare of child

Client understands how to keep child safe

and creates a safe & stimulating home environ

Client understandspotential role of others

in supporting her, and mobilies this support

Client becomes aware of community resources

and accesses these

Client has improved knowledge/ behaviours

in prenatal health

discuss personal health35-40% during pregnancy14-20% during infancy,

10-15% during toddlerhood [15]

discuss maternal role23-25% during pregnancy45-50% during infancy,

40-45% during toddlerhood [16]

discuss lifecourse devt10-15% during pregnancy10-15% during infancy,

18-20% during toddlerhood [17]

discuss environ health5-7% during pregnancy7-10% during infancy,

7-10% during toddlerhood [18]

discuss family + friends10-15% during pregnancy10-15% during infancy,

10-15% in toddlerhood [19]

discuss comm'ty resources% not specified

in licensing req'ts (2-8) [20]

Client plans for futureand achieves goals

re education, employmentand future pregnancies

FN visitsscheduled/structured

as per fidelity requirements [10]

Developing/agreeingreferral pathways

Eligible women recruited by 28th week(60% by 16th week) [6]

75% of those offered programme are enrolled [7]Prod'n of

clear eligibility criteria & referral pathways [4]

Provision of info on eligibility

criteria to key agenciesworking with TG [5]

FNP: Implementation model

FNP: Embedded implementation model

Improved parental

life course

FN visitsscheduled/structured

as per fidelity requirements

discuss lifecourse devt10-15% during pregnancy10-15% during infancy,

18-20% during toddlerhood

discuss comm'ty resources% not specified

in licensing req'ts (2-8)

discuss family + friends10-15% during pregnancy10-15% during infancy,

10-15% during toddlerhood

discuss maternal role23-25% during pregnancy45-50% during infancy,

40-45% during toddlerhood

discuss personal health35-40% during pregnancy14-20% during infancy,

10-15% during toddlerhood

discuss environ health5-7% during pregnancy7-10% during infancy,

7-10% during toddlerhood

Client understands how to

keep child safeand creates a safe

& stimulating home environ

Client becomes aware of community resources

and accesses these

Client plans for futureand achieves goals

re education, employmentand future pregnancies

145 women enrolled

who meet eligibilitycriteria

OUTPUTS SHORT TERM OUTCOMES

Intendeddomain outcomes

(general)

Client has improved health knowledge

and behaviours

Client understands & demonstrates

sensitive/competentcare of child

Uptake of screening services

(CEL 31) [21]

Attends antental appointmentsand classes [22] more natural births [25]

improved knowledgeof health behaviours

and impact on child [26]

Mother taking good care of selfe.g. reduced use alcohol,

tobacco,cannabisgood diet [27]

Better pregnancy outcomes

[28]

knows principles of 'good parenting' [30]

better infant mental health [34]

fewer hazards in homemore safe practices

e.g. use of safety equipment[38]

availability/use of books, toys etc [39]

client engages inhelp seeking behsto avert crisis [41]

involvementof Dads,

other family membersand friends [43]

reduced domestic abuse

[42]

Client understandspotential role of others from personal network

in supporting her, and mobilies this support fewer accidents [40]

More infant HI practices e.g:b-feeding initiat'n

& maintenance (H7); weaning practices (H3)

tooth brushing [29]

decreased child maltreatment [35]

confidence in parenting role [31]

+ve parentingpractices [32]

good bonding

/ attachment [33]

well supportedmother [44]

less anxiety,depression, PND

in mums [45]

referrals to agencies/sources of support [46] use of community

supports/ resources [47] greater interval between pregnanciesfewer unplanned pregnancies

mums know what they want to achievemums on path to meet their goals [48]

Improvedchild health

+ devte.g thriving babies [49]

better prepared for birth [24]

stimulated, alert and responsive

babiesgood language dev't [36]

posseses child safetyknowledge

knows age appopriate ways to stimulate child [37]

uptake of Healthy Start

and use of Vit D [23]

From Having to Using……

We used the logic models to frame monitoring and evaluation of the programme by: – Prioritising key outcomes and assumptions of interest– Prioritise key questions addressing above – Agreeing who would collect and analyse which data, when

and how• Decisions underpinned by considerations of feasibility,

acceptability and data robustness (including how to improve these)

Box code (from logic model) 3 and links to 1 and 2

Logic If the team attend training and are supervised, then they will possess requisite competencies

Question Does team receive the training & support intended & develop req’d knowledge/ skills?

Indicator(s) Who’ll collect data?

Is this a fidelity req’t?

Who’ll analyse?

Any additional considerations?

Proportion of team attending each mandatory course*

% of learning events run*

Self reports of satisfaction and perceived utility/effectiveness of training and learning events.

Frequency of supervision sessions*e.g. for each FN, no of weeks per quarter that timetabled supervision takes place, expressed as a proportion of working/available weeks ie excludes sickness absence

% of required accompanied visits that take place*e.g. no of accompanied visits per FN per 4 months and % of FNs who receive min quota of accompanied visits every 4 months.

Self-reports of feasibility of roles and competence to deliver it (based on job spec and on practice); also whether/how psychologist support worked in practice

Internal

Internal

External (interviews with all FNs and supervisor)

Internal

Internal

External (interviews with FNP)

FR: Attendance at 4 residential training courses

FR: supervisor runs pre and post learning events

FR for weekly supervision

FR for each FN to be accompanied at least once every 4 months

FNP

FNP

ScotCen

FNP

FNP

ScotCen

It will be imperative that the FNP builds in a process of regular review (every 3 months?)in order to address any shortfalls in delivery req’ts.

There would need to be some agreement about what qualifies as supervision e.g. a quick catch up in the corridor?All measures on this page will require good record keeping, submission of records to the administrator within a workable timescaleWe have assumed that the FN’s and supervisor meet person spec in job desc and so this does not need monitored

Box code (from logic model) n/a

Assumption Attrition will be low

Logic The project is only viable if most families participate ANDIf families find the support useful, they will stay engaged

Question Does project meet the fidelity targets for attrition?

Indicator(s) Who will collect data: internal or external evaluation?

Is thisa fidelity requirement/goal?

Who’ll analyse the data in the first instance?

Any additional considerations?

Percentage leaving/dropping out of programme*Calculated as total no having left the programme divided by no enrolled.Implement programme alerts at monthly intervals if feasible.

Internal via UK004B

FR/G:Cumulative prog attrition is 40% or less thro to the child’s 2nd birthdayand is10% or less during pregnancy…..

FNPBut included in ScotCen reports

Box code (from logic model) n/a

Assumption Attrition will be low

Logic The project is only viable if most families participate ANDIf families find the support useful, they will stay engaged

Question Does project meet the fidelity targets for attrition?

Indicator(s) Who will collect data: internal or external evaluation?

Is thisa fidelity requirement/goal?

Who’ll analyse the data in the first instance?

Any additional considerations?

Percentage leaving/dropping out of programme*Calculated as total no having left the programme divided by no enrolled.Implement programme alerts at monthly intervals if feasible.

Acceptability and perceived utility of FN support

Internal via UK004B

External viainterviews with clients/familiesIncl (if possible) a sample of those who drop out/leave)

FR/G:Cumulative prog attrition is 40% or less thro to the child’s 2nd birthdayand is10% or less during pregnancy…..

FNPBut included in ScotCen reports

Form UK004B makes no provision for client leaving because they did not like the programme. Suggest that some extra fields are added to cover broader range of possible reasons for leaving

The M+E framework in action…Outcome: Mother takes good care of self

Logic: If mother takes good care of self, the risk factors for the infant are reduced

Question: Is there evidence that the FNP results in improved knowledge /health behaviours in clients prior to/following birth of baby?

Indicator: Clients’ accounts of what they have learned about risk/protective factors

Topic guide: Have you and your family nurse talked about smoking? What about drinking alcohol? Taking drugs? The food you should eat or not eat during your pregnancy? Keeping the baby safe?

Interview responses

Well I, to be honest I already knew about like drinking alcohol and taking drugs but I never knew about the smoking thing because my gran smoked with all her three kids while she was pregnant and my gran keeps on saying that later on they were fine but..

…because during my past pregnancy I was actually getting ready to drink alcohol again and she convinced me not to because I make a good home for

the baby and the baby’s depending on me and stuff like that

Application

• Scottish context: ensures evaluation relevant to outcomes of interest in Scotland

• Evaluation tools: Identifies data collection relevant to outcomes of interest

• Wider work: informed NHS Lothian on wider maternity services work

Ist report: intake and early pregnancyhttp://www.scotland.gov.uk/Publications/2011/07/28142203/0

Contact: Vikki Milne, victoria.milne@scotland.gsi.gov.uk