EVALUATION PLAN (D3.1) - actondementia.eu · approach is available in Appendices of D3.1 (see Appendix 2). Since then up to the present version, WP3 has been working on evaluation

This document is part of the Joint Action DEM 2 (Grant Agreement

No 678481) which has received funding from the European Union’s

Health Programmer (2014-2020)

Page 1 of 25

EVALUATION PLAN (D3.1)

Prepared by

WP3 - EVALUATION

Maria-Dolors Estrada1,2 MD, PhD & Vicky Serra-Sutton1,2 Sociologist, PhD

1Agency for Health Quality and Assessment of Catalonia (AQuAS, Spain)

2CIBER Epidemiology and Public Health (CIBERESP, as acronym in Spanish)

ACKNOWLEDGEMENTS

WP3 acknowledges the Evaluation Advisory Group (EAG), Impact Assessment Group (IAG) and Program Board (PB) for their contributions to Act on Dementia Joint Action evaluation plan. WP3 would also like to thank the comments and suggestions received from our colleagues from AQuAS and external experts (see Appendix 1 in Appendices of D3.1).

Submission revised version: 23rd February 2018 EAG submission: 28th February 2018

PB submission: March 2018 Final approval: May 2018




Page 2 of 25

TABLE OF CONTENTS

FOREWORD ......................................................................................................................... 3

JUSTIFICATION .................................................................................................................... 6

OBJECTIVES & GOVERNANCE OF EVALUATION ............................................................. 8

EVALUATION APPROACH ................................................................................................... 9

ACTIVITY 1: ACCOMPLISHMENT & QUALITY EVALUATION ............................................10

Milestones and deliverables to be evaluated .................................................................10

Evaluation strategies .....................................................................................................10

1.1. Evaluation of accomplishment applying indicators ...........................................10

1.2. Peer-review of key documents .........................................................................13

1.3. Administering the “Act on Dementia methodological quality tool” and

establishing methodological quality threshold .........................................................13

ACTIVITY 2: IMPACT ASSESSMENT OF ACT ON DEMENTIA JA .....................................15

Evaluation strategies .....................................................................................................15

2.1. Establishing recommended measures for future impact assessment ...............16

2.2. Measurement of perceived global qualitative benefits (impact) & intention to

uptake recommendations........................................................................................19

TIMELINE.............................................................................................................................20

STRENGTHS & CHALLENGES ...........................................................................................22

ABBREVIATIONS ................................................................................................................23

REFERENCES .....................................................................................................................24




Page 3 of 25

FOREWORD

This document is the Evaluation plan of Act on Dementia Joint Action (JA) and presents the

methodological approach to be carried out during the development of the JA, from March 2016

(M1) to February 2019 (M36). This is the first deliverable (D3.1) of Evaluation work package

(WP3). Appendices of the evaluation plan are available in a separate document.

The evaluation plan of Act on Dementia JA presented in this document is structured in different

sections. First section describes the justification of the evaluation of the JA. Objectives and

governance of WP3 is detailed in the second section. The evaluation approach is presented in

the third section. In the fourth section, a detailed description of the evaluation activities within

the project. The last sections show the timeline and the strengths and challenges of this

evaluation plan.

The development of the evaluation plan of Act on Dementia JA began with the revision of the

work plans of the core WPs (WP4-7) by WP3. After verifying that the proposed methodological

approach in each of the four relevant core areas was similar, both in the evidence phase and

in the testing/implementation phase, WP3 decided to develop a common methodological

evaluative proposal.

This proposal was presented to the Evaluation Advisory Group (EAG) and collected the opinion

of strategies and key aspects to be measured via an online survey at the beginning of June

2016. A draft of the evaluation plan was presented during the first meeting of the Program

Board (PB) held at the end of June 2016. The expected date to get the draft plan evaluation

was July 2016 (Milestone 8). Detailed information about the steps to define the evaluation

approach is available in Appendices of D3.1 (see Appendix 2).

Since then up to the present version, WP3 has been working on evaluation plan content

development. In particular, in this development two key challenges have been needed to be

overcome: how to measure methodological quality threshold of complex deliverables and the

impact of assessment strategy of this JA. This has implied communication, participative and

mutual understanding challenges. A final version of the evaluation plan was presented at the

last PB meeting held in September 2017 in Lyon (expected date, August 2016).

The current revised evaluation plan has evolved during this period and, in fact been applied by

WP3 for the evaluation of the first phase of the project (evidence phase). This is the case of

the approach to evaluate accomplishment and peer-reviewing activities and also for the

evaluation of the methodological quality threshold of evidence reports and some dissemination

products.

As a result of evaluation plan assessment by CHAFEA-European Commission representative

members in Act on Dementia JA, and in the frame of the interim report (D1.1) submission, a

revision of the evaluation plan was requested in December 2017. The current version (March

2018), presented in this deliverable is clearer, more precise and gets to the point. It also




Page 4 of 25

establishes a clear cut between testing tasks developed by core WP and impact assessment

of this JA led by WP3, mainly addressed to the impact at global Member States level, in

particular, the willingness to integrate the recommendations drawn from the pilots into the

national dementia strategies and also for future uptake by strategic decision-makers and global

benefits from this JA.

To sum up, there has been a knowledge generation and mutual understanding of what is

expected, and how to assess methodological quality threshold and impact assessment of Act

on Dementia JA. In Figure 1 below, a description of the aim of Act on Dementia, expected

benefits, key areas and main tasks covered within the whole project (evidence,

testing/implementation and coordination together with dissemination and evaluation), related

milestones and deliverables. Figure 1 shows the main tasks of the evaluation plan as well as

the expected deliverables and outputs from WP3.




Page 5 of 25




Page 6 of 25

JUSTIFICATION According to the World Health Organization figures (April 2016), more than 47 million

people around the world have dementia, a number that is expected to double over the

next 20 years. Dementia leads to memory loss, mood changes and the impairment of

cognitive capacities (reasoning and communication), progressively worsening to a

situation in which the autonomy to perform everyday actions and decision-making

capacity are lost. It is therefore essential, among other aspects, to have health and social

services in place that will provide adequate support for people with dementia and their

caregivers. Elements likewise need to be introduced to reduce the variability in the

diagnosis and post-diagnosis process, and in the different care practices in both the

health and social fields, within the context of people with dementia and their carers.

Knowledge and evidence are necessary to provide quality services for people with

dementia and their caregivers. Even if extensive, the situation is not uniform throughout

Europe. The need, then, is to apply the existing knowledge so as to put into practice

recommendations to improve quality of care of people living with dementia and their

carers.

Act on Dementia JA is a collaborative project that aims to promote the implementation in

Member States of coordination actions to improve the situation of people living with

dementia and their caregivers. The project financed by European Commission (Grant

Agreement [GA] Nº 678481, JA DEM-2) is a continuation of the first JA in Dementia (the

European Union JA on Alzheimer’s initiative, ALCOVE). Current JA, Act on Dementia

has been designed to encourage governments, academics and non-profit organizations

to join forces across Europe to tackle problems shared by many European Union (EU)

Member States to advance in knowledge, common understanding and reduce variability

in care and introduce a common shared strategies/policies at micro/meso level (regional/

country/national level).

Four core technical work packages (WP) are the basis of Act on Dementia JA including

two phases: evidence gathering of best practices in key areas and testing of agreed best

practice models and measures for monitoring them. The main four key areas are:

diagnosis and post-diagnostic support (WP4) led by France; crisis and care coordination

(WP5) co-led by Italy and the Netherlands; quality of care in residential care settings

(WP6) led by Norway and; Dementia friendly communities (WP7) led by United Kingdom

(UK)-England.

Multidisciplinary teams are working together, with different academic and professional

profiles, types of organizations and cultural backgrounds (clinical, academic, policy-

making, research, consultancy, advocacy, representatives of people with dementia and

caregivers, among other). The goal is provide decision-making tools that will search to

maximize improvements in the quality of care, health status and quality of life of people

with dementia and caregivers. Moreover, complex phenomena need to be untangled

and understood to potentiate the achievement of Act of Dementia. The development of




Page 7 of 25

decision-making tools is key, not only for policy decision-making but also to apply in the

clinical and care management and at organizational level.

Answering complex goals in the frame of Act on Dementia JA implies being able to

describe benefits and evidence of achievement of its main goals (obtaining evidence in

testing sites of applying best practices in the four core key areas; evidence and

recommendation adoption from non-participating Member States; evidence of stronger

collaboration between Member States in respect to dementia; and evidence of continued

EU prioritisation of dementia).

In addition to the four core WP, Act on Dementia JA also includes three transversal WP:

dissemination (WP2) and coordination (WP1), both led by the Scottish Government (UK)

and evaluation (WP3), led by the Agency for Health Quality and Assessment of Catalonia

(AQuAS, as for acronym in Catalan). AQuAS is a public Catalan governmental agency

from Catalonia in the Health Department (Spain).

Evaluation of this kind of financed European projects has classically implied the

monitoring of accomplishment of key activities, to encourage that goals are achieved an

also for accountability issues. A step forward includes peer reviewing activities and also

the evaluation of methodological quality threshold of key products. Assuring these latest

points is key to increase credibility, soundness and transparency of JA results and

recommendations.

Accountability is key for commissioners and partners implied and especially for citizens.

In the last years, there has been an increased interest in evidencing benefits and impacts

of projects financed by European Commission. In the case of Act on Dementia JA,

specific benefits such as increases in the quality health and social care services and

interventions to promote the empowerment of people with dementia and caregivers and

more integrated interventions are expected.

Impact assessment of JAs can constitute the basis to identify barriers and success

factors in the implementation and transferability of best practices identified and adoption

/uptake of their key recommendations. It should be taken into account, that evidence of

impacts derived from policies & strategic plans are expected at least to show at 5 to 10

years after their implementation. In the case of JAs, that include the development of

evidence-based policy making tools, it’s seems more complex to attribute changes in the

system to the project development and action within the project to increase impact.

Nevertheless, proxies can be described as impacts at medium and long term and also in

qualitative terms to help understand and draw short-term benefits.

To increase the probability of implementing derived recommendations obtained from Act

on Dementia JA for decision-making and transferability across Europe, WP3 proposes

an evaluation strategy focused on two main evaluation activities: 1) evaluation of

accomplishment, peer-reviewing activities of key document and methodological quality

threshold of deliverables and 2) impact assessment of the whole JA.




Page 8 of 25

OBJECTIVES & GOVERNANCE OF EVALUATION

Evaluation objectives

General objective: the evaluation of Act on Dementia JA.

Specific objectives:

1. To support the production of interim and final reports of Act on Dementia JA.

2. To ensure that individual WP deliverables meet an acceptable quality threshold.

3. To evaluate the impact of the JA using appropriate qualitative and quantitative

measures.

Period: from the beginning of the JA to the end (March 2016 to February 2019, M1-

M36).

Evaluation milestones and deliverables: 2 milestones and 4 deliverables.

MS4: Evaluation Advisory Group (EAG) established (M2).

MS8: Draft Evaluation Plan prepared for consideration by Programme Board (M5).

As a result of implementing the evaluation plan (deliverable D3.1), there will be three

evaluation reports (Table 1).

Table 1. Evaluation reports

Deliverable Content Period Deadline

D3.2 Evaluation report of evidence reports M1-M12 M13

D3.3 Evaluation report of testing reports M13-M34 M35

D3.4 Evaluation report of impact Act on Dementia JA M1-M36 M36

Governance of evaluation

AQuAS is who leads the evaluation of Act on Dementia JA (WP3). WP3 is supported by

the Evaluation Advisory Group (EAG), composted by a representative of each WP, and

direct contact with the coordinator of the JA (WP1) and people in charge of dissemination

(WP2). Obviously, for developing all evaluative activities suggested in the evaluation plan

appropriately, a narrow collaboration among all associated partners, collaborating

stakeholders involved in Act on Dementia JA and WP3 team is needed.

Members of the governance body of WP3 are listed in Appendices of D3.1 together with

its role (see Appendix 3).




Page 9 of 25

EVALUATION APPROACH

Evaluation strategies

In the frame of evaluation of Act on Dementia JA, there are two main strategies to

achieve evaluation objectives:

Activity 1: Evaluation of accomplishment, peer review activities of key documents

and methodological quality threshold of individual WP deliverables.

Activity 2: Impact assessment of Act on Dementia JA.

Three deliverables will be produced including the evaluation of evidence reports (D3.2),

the evaluation of testing reports (D3.3) and the impact assessment report (D3.4).

Moreover, a set of additional outputs will be obtain (see Figure 1 in foreword section):

From activity 1:

Final assessment of methodological quality threshold of all evidence reports

(n=5). One evidence report for each core WP except WP4 that has two evidence

reports.

Final assessment of methodological quality threshold of all testing reports (n=4).

One testing report for each core WP.

From activity 2:

A list of agreed measures for future impact assessment of Act on Dementia JA

with a guidance about their future implementation.

Perceived global qualitative benefits & intention to uptake final

recommendations from Act on Dementia JA and their challenges by key

stakeholders.




Page 10 of 25

ACTIVITY 1: ACCOMPLISHMENT & QUALITY

EVALUATION

“Accomplishment and quality evaluation” involves different evaluation activities:

evaluation of accomplishment, peer review activities of key documents and

methodological quality threshold of individual WP deliverables. All of them are related to

main activity 1, one of the two evaluation activities proposed by WP3 for evaluating Act

on Dementia JA.

Milestones and deliverables to be evaluated

According to GA, 10 milestones and 21 deliverables have to be developed within the

three-year project (2016-2019). Nine out of 21 deliverables are reports related to core

WP (WP4-7). Five of nine are “evidence reports” and the remaining four deliverables are

“testing reports”. In the frame of Act on Dementia JA, “Evidence report” and “Testing

report” have the following meaning:

Evidence reports: documents identifying evidence-based best practice models

in four key areas (diagnosis and post-diagnosis support, crisis and care

coordination, quality of care in residential settings and dementia-friendly

communities) for people living with dementia and their carers.

Reports on testing: documents describing implementation plans of the

evidence-based best practice models (evidence reports) for different pilot sites

and presenting testing results.

List of milestones & deliverables of Act on Dementia JA to be evaluated are available in

Appendices of D3.1 (see Appendix 4).


WP3 team proposes different strategies to evaluate accomplishment, peer review and

methodological quality threshold. These activities implies applying indicators to measure

accomplishment, giving feedback or administering a checklist, “Act on Dementia

methodological tool” and other support tools. These indicators as well as the design of

evaluation tools were developed by WP3 in agreement with members of the Evaluation

Advisory Group (EAG) and PB.

1.1. Evaluation of accomplishment applying indicators

A set of eight indicators to evaluate the accomplishment of Act on Dementia JA was

prioritised and defined using an online survey and metaplan consensus meeting with the




Page 11 of 25

EAG and the PB members. The accomplishment will be evaluated for all milestones and

deliverables.

Milestones & deliverables according to their deadlines and main tasks

Timely delivery of milestones and deliverables and content as defined in GA or work

plans contributes critically to the success of the project. In this case, of Act on Dementia

JA.

To evaluate the accomplishment of milestones and deliverables in terms of their

deadlines as well as predefined main tasks, two indicators will be applied taking into

account the complete period (M1-M36). It should be taken into consideration, that

accomplishment of expected deadline means not only the deadline as established in the

GA, but also an approved milestone or deliverable by the PB. Table 2 below shows the

type and title of these two indicators.

Table 2. Titles of indicators for measuring the accomplishment of milestones and

deliverables according to deadlines and main tasks

Process indicator Output indicator

Indicator_1: Percentage (%) of

milestones and deliverables that

accomplished the expected deadline.

Indicator_2: Percentage (%) of

accomplishment with main tasks in

milestones and deliverables.

WP3 will evaluate the accomplishment using the information described in the GA, its

amendment and available in work plans and updates reported by WP team before PB

meetings, the interim report (periodic report) and finally, received milestones and

deliverables.

Indicators 1 & 2 findings are expressed as percentages (%) and then classified

categorically as: full compliance, partial compliance and non-compliance. It should be

noted that in some cases, two components are expected by deliverable (for example, in

the case of D5.1 or D7.1). The standards of reference are the following: Indicator_1: 70%

of milestones and deliverables completed with no more than 2 months delay in relation

to schedule; Indicator_2: 70% of main tasks completed according to GA and its

amendment.

Internal WP meetings

It is important to carry out the internal WP expected meetings defined in GA and work

plans of each WP to increase the communication between partners.

The approach to evaluate the accomplishment of internal WP meetings is based on the

application of two indicators, taking into account the period M1-M12. Table 3 below

shows the type and title of these two indicators.




Page 12 of 25

Internal WP meetings evaluated will be: PB meetings in the case of WP1, Dissemination

Advisory Group (DAG) meetings in the case of WP2, EAG meetings in the case of WP3

and internal plenary meetings in the case of core WP (WP4 to WP7) in evidence phase

(M1-M12).

Table 3. Title of indicators related to accomplishment of internal WP meetings

Type of meeting Process indicator Output indicator

internal WP

meetings

Indicator_3: Percentage (%)

of internal WP meetings in

relation to the expected and

according to time schedule.

Indicator_4: Percentage

(%) of internal WP

meetings carried out

according to expected.

WP3 evaluated the accomplishment of meetings using the same information resources

as indicators 1 & 2 adding information available in AdminProject (Act on Dementia JA’s

management platform).

Indicators 3 & 4 findings are expressed as percentages (%) and then re-categorised as:

full compliance, partial compliance and non-compliance. The standards of reference are

the following: Indicator_3: 70% of meetings carried out in each phase (evidence and

testing) according to expected time schedule; Indicator_4: 90% of expected meetings

carried out in each phase (evidence and testing).

Quality of PB meetings

It is important evaluating the quality of PB meetings. In this sense, this implies to measure

the level of effectiveness in terms of being well planned, executed, and returning feed-

back of agreements as well as to know the global satisfaction of participants and areas

of improvement and needs for the future.

The approach to evaluate the quality of PB meetings is the application of four indicators,

taking into account PB1-3 meetings hold in months 5 to 19. In the case of PB meetings,

the time period of analysis was extended to the beginning of the testing phase to consider

issues related to closing of deliverables from the evidence phase. Table 4 below shows

the type and title of these four indicators.

Table 4. Title of indicators related to quality level of PB meetings

Type of

meeting

Process indicator Output indicator

PB meetings Indicator_5: Minutes of

the PB meetings.

Indicator_6: Number (%)

of participants in the PB

meetings.

Indicator_7: Global satisfaction

(%) of participants with the PB

meetings.

Indicator_8: List of improvement

areas and needs related to the

PB meetings.




Page 13 of 25

WP3 will apply these indicators to evaluate the quality of PB1-3 meetings using the

information available in the updates prepared by each WP before the PB meetings and

interim report. In addition, information available in AdminProject will be considered and

also an ad hoc satisfaction and areas of improvement survey will be sent to participants.

This survey will be sent to all participants by email immediately after the PB meeting by

WP3 team.

Indicators 6 & 7 are reported as percentages and indicators 5 & 8 are collected in

qualitative terms. The standards of reference are the following: indicator_5: list of

participants, agenda and agreements always available and clearly reported (minutes) for

all participants in PB meetings; indicator_6: at least 70% of participants according to

expected (representatives of each WP/partners); indicator_7: 70% of participants are

very satisfied or satisfied with the PB meetings; indicator_8 does not have a reference

standard because it is an open qualitative list.

Chart of all indicators and satisfaction survey are available in Appendices of D3.1 (see

Appendices 5-6 and Appendix 7, respectively).

1.2. Giving feedback of key documents

Peer reviewing activities imply reading and revising draft or advanced versions of key

materials/documents (such as work plans, surveys, presentations, other dissemination

materials) and returning comments and suggestions for improvement to authors and their

working teams. This feedback/peer review was considered a strategy to ensure clarity

and promote quality of Act on Dementia JA products and key documents.

Peer reviewing activities to the interim reports coordinated by WP1 will be carried out. In

the case of WP3 deliverables, the EAG is responsible for peer-reviewing activities related

to WP3 deliverables. PB members are in charge of approving final documents before

their submission to CHAFEA.

1.3. Administering “Act on Dementia methodological quality tool” and

establishing the methodological quality threshold

Formal evaluation of the methodological quality threshold of individual WP deliverables,

specifically evidence and testing reports, is key to increase clarity of the process, validity

and independency of the evaluation. Soundness and potential reproducibility is expected

and in consequence greater impacts.

For evaluating the methodological quality threshold of individual WP deliverables, a two-

step is proposed:




Page 14 of 25

Step 1: administration of the checklist “Act on Dementia methodological quality tool” and

“Return assessment report” to co-authors.

Act on Dementia methodological quality tool: WP3 in agreement with the EAG and PB

designed this methodological quality checklist after revising work plans of Act on

Dementia JA together with the key materials to gain knowledge of key methodological

aspects and key tasks in evidence and testing phases and dissemination tasks. This

checklist was inspired in the AGREE II tool developed by Brouwers M et al. 2010 to

evaluate the methodological rigor and transparency of clinical practice guidelines.

Act on Dementia methodological quality tool is a checklist with six sections and includes

a similar structure and scoring. Each section is aimed at a specific deliverable or set of

deliverables (for example, evidence reports). Section 1 applies to the evidence reports

and includes eleven items classified in four domains. Section 2 applies to the testing

reports and includes only one domain with nine items. Section 3 refers to the

dissemination plan and dissemination reports (interim and final) including a domain and

five items. Section 4 refers to Act on Dementia website including a domain and eleven

items. Section 5 refers to the leaflet and includes a domain and two items and finally,

section 6 refers to the Layman version of the final report of Act on Dementia JA and

includes a domain and five items. Each section includes a global evaluation score of the

methodological quality of the deliverable. The score ranges from 1 to 7 (1 implies the

lowest possible methodological quality and 7 the highest methodological quality). In

some cases, such as for Evidence or Testing sections, composite items are included.

This checklist, additionally, allows reporting on qualitative peer-review comments in

structured and complements or helps to justify the above-mentioned quantitative quality

score. The checklist has some instructions with the intention to homogenise its

administration. The complete tool is available in the appendices of D3.1 (see Appendix

8).

Return assessment report: WP3 developed this return report to inform on the evaluation

after applying the “Act on Dementia methodological quality tool” to co-authors. The return

assessment report has three parts: part 1- global evaluation of methodological quality of

deliverable and comments/ areas for improvement in general terms; part 2-specific

evaluation by each assessed domain of the above mentioned checklist, with a specific

score (1 to 7) for each domain; and part 3-additional considerations.

The process starts when the leader or co-leaders of each individual WP deliverable send

an advanced version to WP3 by email. WP3 applies the checklist and elaborates the

corresponding return assessment report that it is returned to authors/co-authors. Two or

three members of WP3 team administer the checklist and writes the return assessment

report to be sent to leaders of the deliverable.




Page 15 of 25

Step 2: establishing the methodological quality threshold of individual WP deliverables

through “Final assessment report”.

Final assessment report: this assessment report is defined as a public document to be

published in CHAFEA and EU portals and Act on Dementia JA website and informs of

the methodological quality threshold of each assessed deliverable, as well to summarize

all the appraisal process followed. Each deliverable obtains a global methodological

quality threshold (from 1 to 7, 7 highest methodological quality). Additionally, several key

elements of each deliverable will be considered to summarize the level of methodological

quality in qualitative terms ([+++] represents the highest quality and [+] the lowest).

WP3 team agrees on the final quality threshold and content of the “Final assessment

report” and sends to co-authors. After this, the evaluation process is finished from WP3

point of view. It should be noted that deliverables have to be approved by PB members.

ACTIVITY 2: IMPACT ASSESSMENT OF ACT ON

DEMENTIA JA

Impact assessment is the second main activity in the evaluation WP.


WP3 proposes two approaches for the evaluation of impact of Act on Dementia. The first,

is addressed to establish a list of recommended measures for future impact assessment

(at short, medium and long terms). This activity implies defining a theoretical framework

together with a proposal of impact measures to be agreed on by stakeholders across

Europe.

The 2nd activity related to impact assessment is to apply qualitative techniques to collect

the opinion of strategic stakeholders at the final conference of Act on Dementia JA. The

main subjects to be collected will be: a) perceived global benefits (impacts), b) the

intention to uptake key recommendations and c) barriers and facilitators in the

implementation of these recommendations at meso and macro level. This information

will be complemented with context from national health/social plans and policy

documents in the field of dementia and main core areas of Act on Dementia JA. The

later, will be triangulated with the information collected through semi-structured

questionnaires and qualitative interviews to add rigor and understanding of phenomena

been studied.




Page 16 of 25

2.1. Establishing a list of recommended measures for future impact assessment

Theoretical framework

Fixing the conceptual framework for the definition of measures for future impact

assessment of Act on Dementia JA will aid in scoping and defining relevant concepts to

take into account along the process. The impact assessment framework of Act on

Dementia JA (Figure 2) has been proposed based on ad-hoc literature review carried

out by WP3 and consultation to key experts in the field and taking into account GA and

its amendment and inputs from members of EAG and PB, CHAFEA, European

Commission, DG Santé, International Consortium of Health Outcome Measurement

(ICHOM), Policy Innovation Research Unit (PIRU, London School of Hygiene & Tropical

Medicine), JA CHRODIS.

This framework has four components as follows:

Component 1: research impact framework model published by PIRU (Kuruvilla

S, Mays N, et al, 2006).

Component 2: overall expected benefits of Act on Dementia JA (GA and its

amendment).

Component 3: key specific expected benefits for core area of Act on Dementia

JA (GA, its amendment, work plans and deliverables).

Component 4: working model of ICHOM and specific standard set in dementia

(http://www.ichom.org).

Figure 2. Act on Dementia JA impact assessment framework




Page 17 of 25

A preliminary list of concepts/titles of measures for future impact assessment have also

been proposed by WP3 (in appendices D3.1, see Appendix 9). The proposal has been

useful to explore their content validity and adequacy of this framework and it usefulness

at practical level in Act on Dementia JA. This proposal of measures needs to be agreed

on by members of this JA and also by a wide and multidisciplinary groups of stakeholders

across Europe applying consensus techniques.

Sampling and participants

A first group of experts closely connected to Act on Dementia JA (n= 50, expecting 50%

of response rate) will be asked to participate. They will be asked to score and prioritize

in an online platform a selected list of impact measures (approximately 25-30) for future

impact assessment of Act on Dementia JA. The scores of this expert group will be used

as a reference for another more extensive group that will be participating in a second

wave to score the same list of measures.

A second more extensive group of stakeholders across Europe will be asked to

participate in the scoring and prioritization of the proposed and selected list of measures

voted by the small group mentioned above. The strategy to reach a wide range of

stakeholders from member and non-member States will be through scientific societies,

health and social ministries and organizations representing the voices of people living

dementia and caregivers, also European Commission/ CHAFEA/ Alzheimer Europe and

DG Santé portals. To assure confidentiality and data protection, a link to the project will

be included in the websites of key organizations and newsletters asking to contribute to

this consensus exercice (300 participants are expected to an online Delphi).

Expected participants characteristics: people with dementia and caregivers, public health

and social decision-makers, planners, managers, technicians and researchers in the

field of dementia, chronicity, integrated care, health service research, quality

management and improvement, information systems, professionals involved in the care

of people living with dementia and caregivers (primary care physicians, neurologists,

psychiatrists, geriatrists, nurses, psychologists, social workers, among other profiles).

As mentioned above, participants will not be asked nominally but key institutions will be

contacted to send a call to participate in this consensus activity. A link to access the

consensus online platform and short note informing on the goals, and call to participation

will be included in different websites, periodical newsletters, twitters and other social

networks. Once the participants enter the platform anonymously, participant will be able

to describe their profile, country where they work and also type of institution, gender and

age.

Data collection and type of consensus

Participants will be asked to vote the relevance of the proposed list of impact measures

for future impact assessment using an online web based survey in a platform easy to

answer and user-friendly. Data collection will be carried out by WP3 in collaboration with

a subcontracted consulting company, using the Health Consensus online platform from

Abril 2018 to June 2018 (M26-M28).




Page 18 of 25

Health Consensus is a web-based collective intelligence tool inspired in the Delphi

method. Health Consensus tool allows wide and extensive multidisciplinary participation,

allowing hundreds of participants to come along regardless of geographic and

professional limitations and score in a quantitative way measures or concepts1. This tool

has been applied in several projects in the process of development and consensus of

indicators in health service research and decision-making contexts at meso and macro

levels.

A potential list of approximately 20-30 future impact measures will be included in Act on

Dementia Health Consensus. Measures will be classified according to their type

(process, output or outcome), component of Act on Dementia JA impact assessment

framework, and main areas of interest.

In an initial instruction screen, a brief description of the aim of the project and content

will be included. In another screen the list of impact measures will be included. Each

participant will be asked three questions for each potential impact measure included in

this list and will be asked to answer them in a score range from 1 to 7 (in Appendices

D3.1, see Appendix 9): 1) Do you consider the measure relevant to assess impact of

Act on Dementia? 2) Do you consider this impact can be achieved within the next 5

years? and, 3) within the next 10 years? Once each measure is vote a chart is showed

on the screen describing the number of participants up to that moment, and a raw median

score for each specific measure. Participants will be able to change their score if they

consider it necessary. This activity takes approximately 15 minutes.

Analysis of consensus & final recommended measures for future impact

assessment:

The online platform (Health Consensus) generates automatic descriptive statistics for

each related measure (global mean, median, standard deviation and interquartile range

and number of participants). For each measure these descriptive statistics can be shown

in global or stratified by subgroups such as gender, age, profile or country. High

consensus with each measure will be established when 70% of participants a score

between 6-7 (7-being the highest score in the above mentioned score range from 1 to

7).

A final list of recommended impact measures for short, medium and long terms for future

impact assessment of Act on Dementia JA will be described taking into consideration the

following issues:

Act on Dementia impact framework.

Results from Act on Dementia JA Health Consensus

1 Martí, T.; Monguet, J.M.; Trejo, A. Collective health policy making: a priority setting and policy

monitoring application in the Catalan National Health System. A: Collective Intelligence.

"Collective Intelligence Conference, June 10-12, 2014 at MIT". Boston: 2014, p. 1-5.




Page 19 of 25

Experience of implementation of testing measures and barriers/facilitators (as

recommendations from core WP4 to WP7 in the pilot sites)

Final review of key literature to reinforce content validity and robustness by WP3

of selected measures.

Once agreed on the measures to be included in the final proposal for future impact

assessment, a chart for each indicator will be development. Each chart will include

information about justification, formula, and standard of reference, level of agreement,

robustness and validity as well as sources for its calculation.

Financial considerations:

A specific budget was approved in the GA in relation to subcontracting activities of WP3,

specifically for the field work analysis of consensus.

2.2. Measurement of perceived global qualitative benefits (impact) & intention to

uptake recommendations in global terms within Act on Dementia JA

Act on Dementia JA will provide decision-making tools that will search to maximize

improvements in the quality of care, health status and quality of life of people living with

dementia and caregivers. Global key strategic recommendations to promote evidence

based best practice models suggested within Act on Dementia JA and their intention to

be uptaken, collaboration and prioritization of dementia in Europe will be obtained.

When complex phenomena want to be explored and collecting the opinion and

experiences of key stakeholders is relevant, qualitative techniques can reach where

other methods cannot reach (Popey C, et al British Medical Journal,1995). A

phenomenological qualitative theoretical approach will be applied.

Sampling and participants

A theoretical sample of 25-30 stakeholders (key informants) across Europe will be

defined. The goal is to collect the opinion from at least two representatives of health and

social policy-making sphere at national and regional level from each main partners’

country in this Joint Action (Bulgaria, France, Greece, Italy, Norway, Poland, Spain-

Catalonia, UK-England and Scotland), one person representing each collaborating

stakeholder’s country (Cyprus, Germany, Luxemburg, Malta, and Portugal), and finally

one person representing each of the following organizations: Alzheimer Europe,

CHAFEA, European Commission, DG Santé, World Health Organization and

International Consortium of Health Outcome Measurement. The leader of a similar JA

such as CHRODIS+ will be also asked to participate in this part of impact assessment.

The characteristics of these stakeholders include: regional/ national strategic health,

social and wellbeing planning/ or in the dementia field, chronicity and patients and

caregivers advocate representative groups. The basis for selecting these key informants

will be national/ country-level view, nevertheless special emphasis to collect meso/ micro

level regional best practices in policy-making initiatives and points of views will be made.




Page 20 of 25

Contacts will be made by WP3 including a support letter signed by coordination team

and key members of PB depending on the country.

Data collection and qualitative techniques

Semi-structured questionnaires will be sent by WP3 to key informants by email. The goal

is to collect their opinion about context country/ region dementia plans, perceived

benefits of Act on Dementia JA and intention to uptake their key recommendations some

weeks before the final conference meeting. During this final meeting, a face to face

interview will be carried out to talk about these subjects in more depth. WP3 will send a

short summary document (power point) with key recommendations derived from Act on

Dementia as support material to answer the semi-structured questionnaires. Additionally,

key strategic national material (for example dementia cares plans) will be requested by

WP3 to key informants before the interviews.

A group of 5-6 researchers will carry out face-to-face interviews during the 2-day final

conference of Act on Dementia JA. If necessary they will travel and meet by convenience

to interview all selected key informants. Ideally, all interviews will be face to face and will

take between 1:00 to 1:30 hours.

Data analysis Data analysis will coordinated by WP3 with support of the qualitative consultants.

Relevant material such as strategic plans and key documents or presentations, circuits,

will be taken into account to help to contextualize each experience. All qualitative

interviews will be audio-taped and notes will be collected during each interview. Each

interview will be “emptied” and main preliminary insights will be presented in an individual

report for each interview. Content and discourse analysis will be carried out manually by

two-three researchers. Triangulation of main findings from questionnaires and interviews

will be made and among analysists. Members of WP3 will be assuring validity of the

process using a methodological quality checklist for qualitative studies and will be in

charge of writing the final report of impact assessment of Act on Dementia with the

support of subcontracting senior consultants. Before triangulating main global findings,

each emptied interview and preliminary findings will be sent to each key informant for

their validation.

Financial considerations

A specific budget approved in the GA was defined to subcontract the services from an

expert consultant company to give support in the impact assessment of Act on Dementia

JA. Specifically for field work and analysis of qualitative techniques.

TIMELINE

Figure 3 shows the schedule of the WP3 activities throughout the 3 years-project

according to the milestones and deliverables of the Act on Dementia JA. It also shows

the expected meetings with EAG and IAG (at least, once before each PB meeting).




Page 21 of 25

Figure 3. Timeline of the WP3 activities

WP3 milestones & deliverables Mar to Dec 2016 2017 2018 2019

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

EAG established (MS4) x S

Draft evaluation plan (MS8) x x x x S

Evaluation plan (D3.1) x x x x x S

Evaluation report-evidence reports (D3.2) x x x x x x x x x x x x S

accomplishment x x x x x x x x x x x x x

peer-reviewing x x x x x x x x x x x x x

quality evaluation x x x x x x x x x x x x x

Evaluation report-testing reports (D3.3) x x x x x x x x x x x x x x x x x x x x x x S

accomplishment x x x x x x x x x x x x x x x x x x x x x x x

peer-reviewing x x x x x x x x x x x x x x x x x x x x x x x

quality evaluation x x x x x x x x x x x x x x x x x x x x x x x

Evaluation report-impact assessment (D3.4) x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x S

Health Consensus online platform x x x

questionnaires & interviews x x

EAG meetings m m m m m m m

IAG meetings m m m m m m m

Legend S: deadline milestone & deliverable submission for approval by PB Yellow: PB meetings Red: WP3 milestones & deliverables deadlines m: teleconferences meetings expected EAG: Evaluation advisory group IAG: Impact assessment group x: operative related activities within a month work time




Page 22 of 25

STRENGTHS & CHALLENGES

The evaluation plan as any other proposal is complemented by strengths and challenges,

all established a priori by WP3 (Table 5).

Table 5. Strengths & challenges of the evaluation plan

STRENGTHS

CHALLENGES

Accomplishment of milestones & deliverables

Only two easy-to-use indicators. Propose and agree on a short list of accomplishment measures and differentiate tracking (WP1) from evaluation (WP3).

Accomplishment of internal WP meetings

Only two easy-to-use indicators. Propose and agree on a short list of accomplishment measures and differentiate tracking (WP1) from evaluation (WP3).

Quality level of PB meetings

Only four easy-to-use indicators. Satisfaction survey also allows the detection of needs and areas of improvement for future meetings.

Dependent on the response rate and not always areas of improvement could be considered or agreed as feasible (face to face meetings and skype).

Peer reviewing activities of key documents

Participative approach and sense of belonging to JA DEM-2 team.

Workload when non-advanced versions are circulated and procedure instructions and deadlines are lacking.

Administering the methodological quality

tool and establishing methodological quality

threshold

Participative approach and systematic and transparent process.

Absence of specific checklist Design and content consensus of a new checklist

Methodological quality threshold

assessment

Offers systematic and transparency process of evaluations

Implies an effort in a mutual understanding of the threshold

Definition & consensus of impact measures for

future impact assessment Act on Dementia JA

Participative approach, generates collective intelligence and robust consensus of measures.

Response rate can be lower when topics are not interesting for stakeholders or are not in the political/context agenda.

Measurement of perceived global

qualitative benefits (impact) within Act on

Dementia JA

Being able to measure benefits (impact) and understand them.

The participation of key informants (decision-making sphere) with a global view of national and regional perspectives. .

Intention to uptake recommendations in

global terms (within Act on Dementia JA)

Is useful and complements the information on perceived benefits, especially to help to define future strategies to improve recommendations from Act on Dementia.

Can be challenging when small time is left between the final formulation of recommendations and collecting the opinion on intention to uptake them.




Page 23 of 25

ABBREVIATIONS

ALCOVE The EU JA on Alzheimer’s initiative

AQuAS Agency for Health Quality and Assessment of Catalonia, Spain

D Deliverable

DAG Dissemination Advisory Group

DEM 2 Dementia 2

DFC Dementia Friendly Communities

DG SANTE Directorate-General for Health and Food Safety

EAG Evaluation Advisory Group

EU European Union

GA Grant Agreement

IAG Impact Assessment Group

JA Joint Action

M Month

MS Milestone

PB Programme Board

UK United Kingdom

WP Work package




Page 24 of 25

REFERENCES

Addressing Chronic Diseases and Healthy Ageing Across the Life Cycle. CHRODIS project. http://chrodis.eu/about-us/.

ALCOVE evaluation plan, Slovenska Akademia Vied – Neuroimmunologicky Ustav, French National Authority for Health and ALCOVE group, www/alcove-project.eu

Alzheimer Cooperative Valuation in Europe. ALCOVE project. http://www.alcove-project.eu/index.php?option=com_content&view=article&id=28&Itemid=141. Arakelian E, Gunningberg L, Larsson J. Job satisfaction or production? How staff and leadership understand operating room efficiency: a qualitative study. Acta Anaesthesiol Scand 2008;52:1423-28. Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ : Canadian Medical Association Journal. 2010;182(18):E839-E842. doi:10.1503/cmaj.090449

Campbell SM, Braspenning J, Hutchinson A, Marshall MN. Improving the quality of health care: Research methods used in developing and applying quality indicators in primary care. BMJ 2003;326;816-819.

Elaborating a Dissemination Plan [web]. CHAFEA (European Commission); 2016 [searched November 2016]. Available in: http://ec.europa.eu/chafea/management/Fact_sheet_2010_06.html

EUnetHTA Joint Action 3 (2016 - 2020). http://www.eunethta.eu/activities/joint-action-3/jointaction31/eunethta-joint-action-3-2016-2020 European policy health impact assessment - a guide (2004). http://ec.europa.eu/health/ph_projects/2001/monitoring/fp_monitoring_2001_a6_frep_11_en.pdf Ex-post evaluation of the 2nd Health Programme 2008-2013. Final report. European Commission. https://ec.europa.eu/health/sites/health/files/programme/docs/ex-post_ev-hp-2008-13_final-report.pdf Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009 Jun;26(2):91-108. doi: 10.1111/j.1471-1842.2009.00848.x. https://www.ncbi.nlm.nih.gov/pubmed/19490148 Greenhalgh T, Annandale E, Ashcroft R, Barlow J, Black N, Bleakley A, et al. An open letter to The BMJ editors on qualitative research. BMJ. 2016 Feb 10;352:i563. doi: 10.1136/bmj.i563. HAS/ ANAP. Deliverable 4. Report on standard for key inputs. WP5. Day Safe project, 2013. Available in: http://www.daysafe.eu ICHOM. Dementia data collection reference guide. Version 1.0. Revised: February 1st, 2016. http://www.ichom.org/medical-conditions/dementia/ Improving patient safety of hospital care through Day Surgery. DAYSAFE project. http://aquas.gencat.cat/ca/projectes/innovacio_projectes_internacionals/altres_projectes_internacionals/daysafe/ Joint actions. Quality Action: Evaluation Issues.http://ec.europa.eu/chafea/health/actions.html; http://ec.europa.eu/chafea/documents/health/quality-action-evaluation-issues_en.pdf

http://chrodis.eu/about-us/

http://alcove-project.eu/

http://www.alcove-project.eu/index.php?option=com_content&view=article&id=28&Itemid=141

http://www.alcove-project.eu/index.php?option=com_content&view=article&id=28&Itemid=141

http://ec.europa.eu/chafea/management/Fact_sheet_2010_06.html

http://www.eunethta.eu/activities/joint-action-3/jointaction31/eunethta-joint-action-3-2016-2020

http://www.eunethta.eu/activities/joint-action-3/jointaction31/eunethta-joint-action-3-2016-2020

http://ec.europa.eu/health/ph_projects/2001/monitoring/fp_monitoring_2001_a6_frep_11_en.pdf

https://ec.europa.eu/health/sites/health/files/programme/docs/ex-post_ev-hp-2008-13_final-report.pdf

https://ec.europa.eu/health/sites/health/files/programme/docs/ex-post_ev-hp-2008-13_final-report.pdf

https://www.ncbi.nlm.nih.gov/pubmed/19490148

http://www.daysafe.eu/

http://www.ichom.org/medical-conditions/dementia/

http://aquas.gencat.cat/ca/projectes/innovacio_projectes_internacionals/altres_projectes_internacionals/daysafe/

http://aquas.gencat.cat/ca/projectes/innovacio_projectes_internacionals/altres_projectes_internacionals/daysafe/

http://ec.europa.eu/chafea/health/actions.html

http://ec.europa.eu/chafea/documents/health/quality-action-evaluation-issues_en.pdf




Page 25 of 25

Kuruvilla S, Mays N, Pleasant A, and Walt G. Describing the impact of health research: a Research Impact Framework. BMC Health Services Research 2006, 6:134. https://goo.gl/gZh7GF. Martí T, Monguet JM, Trejo A. Collective health policy making: a priority setting and policy monitoring application in the Catalan National. Health System. Collective Intelligence 2014. https://goo.gl/aZRoGs. Monguet JM, Trejo A, Martí T, Espallargues M, Serra-Sutton V, Escarrabill J. Assessment of chronic health care through and Internet consensus tool. IGI Global; 2015. Muir Gray J A. Evidence-based healthcare: how to make health policy and management decisions. London: Churchill Livingstone; 1997. Plochg, T & van Zwieten, M. (2002). Guidelines for Quality Assurance in Health and Health Care Research: Qualitative Research. Academic Medical Centre. Pope C, Mays N. Qualitative Research: Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ 1995; 311 doi: https://doi.org/10.1136/bmj.311.6996.42 Porter ME. What Is Value in Health Care?. N Engl J Med 363;26. NEJM.org december 23,2010. http://www.nejm.org/doi/pdf/10.1056/NEJMp1011024 Report from the commission to the European parliament, the council, the European Economic and social commitee and the committee of the regions. Ex-post evaluation of the 2nd Health programme 2008-2013 under Decision No 1350/2007/EC establishing a second programme of Community action in the field of health (2008-2013); 2016. Sampietro-Colom L, Lach K, Pasternack I, Wasserfallen JB, Cicchetti A, Marchetti M, et al. Guiding principles for good practices in hospital-based health technology assessment units. International Journal of Technology Assessment in Health Care, 31:6 (2016). Serra-Sutton, V et al 2016. Perceived benefits of working in a collaborative model of social and health care in Catalonia: what do professionals involved experience? International Journal of Integrated Care, 16(6):A149, pp. 1-8. http://www.ijic.org/articles/abstract/10.5334/ijic.2697/ Shiffman RN, Dixon J, Brandt C, Essaihi A, Hsiao A, Michel G, et al. The Guideline Implementabiliy Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak [Internet]. 2005 Jul; 5:23 [consultado 15/12/2016]. Available in: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1190181/pdf/1472-6947-5-23.pdf Sustaining Development in Early School Education. SUSTAIN project. http://www.sustain-project.eu/

UbiK media, Ramírez Ruiz JM. Proyecto TRAM-S (Modelo de Traslación de Resultados de la Investigación Médica a la toma de decisiones en Salud). Plan Básico de diseminación de guías de práctica clínica (TRAM-S); [searched November 2016]. Available in:

http://trams.rdi.uoc.edu/wp-content/uploads/2016/02/TRAMS-Plan-Basico_ES_vFinal.pdf.

https://goo.gl/gZh7GF

https://goo.gl/aZRoGs

https://doi.org/10.1136/bmj.311.6996.42

http://www.nejm.org/doi/pdf/10.1056/NEJMp1011024

http://www.ijic.org/articles/abstract/10.5334/ijic.2697/

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1190181/pdf/1472-6947-5-23.pdf

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1190181/pdf/1472-6947-5-23.pdf

http://www.sustain-project.eu/

http://www.sustain-project.eu/

http://trams.rdi.uoc.edu/wp-content/uploads/2016/02/TRAMS-Plan-Basico_ES_vFinal.pdf

Documents

EVALUATION PLAN (D3.1) - actondementia.eu · approach is available in Appendices of D3.1 (see Appendix 2). Since then up to the present version, WP3 has been working on evaluation